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Purposes of evaluation: Student 
evaluation and program evaluation 
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Sources of evaluation information 
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What the three STEPS in 
TASK "J" accompl ish 
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Cyclical nature of program tryout 
and revision 
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Four interrelated elements in the 
tryout and revision process 
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Hew much program material 
to schedule for tryout 


7 



1 



J. 



TWO MAJOR PURPOSES IN USING EVALUATION PROCEDURES 



IDENTIFICATION 
MATRIX 



PURPOSES 


Assessing the 
ADEQUACY OF THE 
INSTRUCTIONAL PROGRAM 


Assessing the 
LEVEL OF STUDENT PROFICIENCY 

* 




-An asseament performed only on 
a limited number of occasiortQ: 

• 'DuriTig the preparation of the 
instructional program 


-An assessment performed on a , | 
routine basis every time the 
instructional program is given 




••Immediately after the program 
has been completed 




CRITERIA 


'-AsGeejment is discontinued when 
student performance based on the 

standard 






-Assessment is used to identify 
weaknesses (alloxoing those 
portions of the program which 
result in errors to be revised) 


-Assessment is used to identify 
levels of student proficiency as 
a basis for decisions: 

• *To allow students to go on to 
new portions of a program 

OR 

instructional sequences 








^Assessment is based on: 


-Assessment is based on: 




^•Student perfomance on the 
instnwtional program 


• 'Student performance on criterion 
and sub-criterion tests 




Student performance on 
criterion and sub-criterion 
tests 






••Student performance on 
diagnostic tests 




[ 


** Student responses to diagnostic 
probes during interviews 
(during program development) 

1 





^I^Only evaluation for purposes of "assessing the adequacy of instructional 
programs** Is treated In this volume devoted to TASK "J," 
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J. 

IDENTIFICATION 
MATRIX 



THREE SOURCES OF INFORMATION USED AS A BASIS 
FOR IDENTIFYING PROGRAM WEAKNESSES AND FOR REVISING THE PROGRAM 



SOURCES OF 
INFORMATION 



Student performance on 
PROGRAM PROBLEMS 



-Student performance 
on progrcon problems 
or tasks is used to 
identify errors 
CRITERIA occurring on specific, 

**Ppogr*am problems 

• ^Program procedures 
or tasks 

"To the extent that a 
problem or task is 
restricted to one 
particular component 
skilly performance on 
that problem can be 
diagnostic of a 
failure to acquire 



Student responses to 

INTERVIEW PROBES 

about performance on 
the program 



-W^en a problem 
solution is dependent 
on more than one 
skilly (and an error 
is therefore not 
diagnostic of a 
particular learning 
failure) J the use of 
interview probes can 
lead to the identifi- 
cation of: 

**The nature of the 
error made 

* *The possible source 
of the error 



Student performance on 
TESTS 



Student performance 
on tests (whether 
criterion J 
sub-criterion^ or 
diagnostic) is used 
to identify the 
adequacy of 
instructional 
responses to teach: 

Criterion behaviors] 

• •Sub-criterion 
behaviors 



TO BE 








COVERED IN: 


Sub-STEP J. 1 . 1 


Sub-STEP J.l .1 


Sub-STEP J. 1 .2 
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J. 

IDENTIFICATION 
MATRIX 



WHAT IS PERFORMED IN THE THREE "STEPS" OF TASK "J" 



STEPS 


J.l 


J. 2 


J. 3 




INFORMAL TRYOUT 


DEVELOPMENTAL TRYOUT* 


FIELD TESTING** 



CRITERIA 



-The informaZ tryout 
of the program with 
a small sa^rrple of 
students from the 
target population 

-The revision of the 
program based on 
student performance 
on program problems 
and tasks 



-The administration of 
the program to a 
representative sample 
of the target 
popu lation: 

•*The identification 
of program problems 
which result in a 
relatively high 
percentage of errors 

-The rdminiotration of 
tests to determi7ie the 
level of proficiency 
achieved by students 
for the criterion and 
sub-criterion 
behaviors 

-The revision of the 
program based on 
sequences identified 
as not having 
produced an adequate 
proficiency leve I 



-The administration of 
the program to a 
large sample of the 
target population as 
a means of: 

• • Providing 

specifications of 
the program 's 
capability of 
producing a 
pre-de termined 
level of 
profici eKoy 



TO BE COVERED IN: 


STEP J.l 


STEP J. 2 


STEP J. 3 


ON PAGE; 


9 


89 


221 



*often referred to as "formative evaluation" 
**often referred to as "sutnmative evaluation" 
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J. 

MATRIX 



THE CYCLICAL NATURE OF THE TRYOUT AND REVISION PROCESS 



STEPS 



CRITERIA 



J.I 

INFORMAL TRYOUT 



'Infomal developmental 
try out is performed 
cyclically : 

••One or tDo students 
take the program, 
and program problemB 
on which they make 
errors are revised 

••An additional one 
or tb)o students 
take the revised 
program; the errors 
they make on the 
program become the 
basis for a further 
revision 

^Additional informal 
tryout cycles may he 
used depending on 
time/money considera- 
tions a}id on hew well 
the program is working 



J. 2 

DEVELOPMENTAL TRYOUT 



»i4 tryout of the 
program with a 
representative sample 
of the target popula- 
tion is performed 
cyclically: 

• •Approximately 25-50 
students take the 
program arid 
associated tests; 
revisions in the 
program are made 
baaed on program 
errors and test 
scores 

••An additional 
sample takes the 
revised program; 
program errors and 
test errors are 
noted; if the 
predetermined 
standards are still 
not mety an 
additional revision 
is made, and the 
revised version is 
again tried out 

-The tryout and 
revision cycle ends 
only when a 
predetermined standard 
is met 



J. 3 

FIELD TESTING 



'The final evaluation 
of the program is not 
used as a basis for 
revision 

'The final evaluation 
with a large sample 
of the target popula- 
tion (a sample of 
approximately 200- 
500) is condicted to 
provide specifica- 
tions for the program 
(if it is to be given 
wide distribution) 

-Only test results 
(not program problem 
results) are likely 
to be reported 
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J. 

DECISION 
MATRIX 



HOW MUCH PROGRAM MATERIAL TO SCHEDULE FOR TRYOUT* 



CONDITIONS 


MORE THAN ONE 
criterion behavior 

covered In an 
instructional hour 


ONE 

criterion behavior 

covered in an 
instructional hour 


It require'^ 
TWO OR MORE 
INSTRUCTIONAL HOURS 

to cover a 
criterion behavior 


ACTION 
TO TAKE 


'Schedule the tryout of all the instructional 
material it requires to administer 
(approximately one hour) 


-Schedule (in one 
sitting) the tryout 
of all the 
instructional 
material associated 
with sub'Critenon 
behaviors covered in: 








••One instructional 
hour 

OR 

••Two or three 
instructional hours 
(at a maximum) 








^Schedule as many 
separate tryout 
periods as required 
to ,covqr the 



*Thft subsequent discussion is concerned with the tryout of instructional materials 
covering a single criterion behavior and its associated sub-criterion behaviors. 
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STEP 0.1 



Conduct an Informal tryout of the Instructional program and revise the program 
when necessary flf^ 



J. 1.1 



Administer criterion test (and sub-criterion test) as a "before" 
test to a small sample of the target population. 



0.1.2 



Administer the program to one student at a time, and, when errors 
are committed, probe for the source of the problem. 



J. 1.3 



Interpret the nature of the learning failxire. 



0.1.4 



Make revisions in the instructional program based on identified 
program weaknesses. 



cycle of Sub^STEPS J .1.1, J. 1.2, J. 1.3, and J. 1.4 should be 
repeated as many times as is feasible within schedule and budgetary 
constraints. Reduced frequency of student errors indicates that 
there is no need for further recycling of informal tryout and revision. 
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STEP 



INPUT 



First version of 
instructional program 
+ 

criterion test 
+ 

sample audience 



OVERVIEW 




ACTION 



(Cyclically) 
Administer program, 
identify weaknesses, 
and revise program 



OUTPUT 



Revised instructional 

program ready for 
developmental testing 



JJ.l 



J. 1.2 



J. 1.3 



Target audience 



Select 



representative 
sample 
and administer 
sub-criterion and 
criterion tests 
to them 

ii 



Sample of students 
selected for 
informal tryout 



IV 



Administer 
instructional program: 

-Observe and make a 
record of errors, AND 

-Probe for type and 
source of failure v 



Student performance 
on program 
and his responses 
to diagnostic probes 

vii 



Identify the 
type of learning failure 
which has occurred 



jriiij 



JJ.4 



Identified weakness 
in program 



Revise the program 



xi 



Sample of identified j 
"low" proficiency 
+ 

identification of 
portions of 
criterion behavior in 
its repertoire iii 



Identification of: 

-Program problems on 
which errors were made 

-Type of learning 
failure 

-Sources of the vi 
failures 



Identification of types 
of learning failures 
which have occurred 



ix 



Revised program 

(ready for 
further tryout) 



xii 
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STEP 



PAGE INDEX 



CRITERIA FOR 
IDENTIFYING INPUTS 



ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRJX: Information 
obtained from 

"before" aiW "-if ter'' 

-MATRIX: Properties 
of adequate tryout 




-MATRIX: Adequacy 
of sample 
selection and use 
of pre-instruction 
test results . . 21 


SUMMARY OF 

PROCEDURES ... 20 




|-MATRIX: What 
I information 

observati on 

provides • • . . 31 
-MATRIX; When 

probes are 


-MATRU: Priorities 
in types of probes 

-MATRiX: Types of 
probes 

to use .... .36-'i2 

-r.ATKlX: Timing 
the use of 


-MATRIX: Adequacy 
of observatfonal 
and probing 
process . . . • ^7 


(Optiona] ) 
FORM J. 2(1) 

SUMMARY OF 

PROCEDURES . . . A6 




-MATRIX: Types of 
student 

difficulties .... 52 
-MATRIX: Symptoms of 

d i f f i ru 1 1 ies wi tb 

program format ... 53 
-MATRIX: Symptoms of 

d i f f i cul t i cs wi th 

program content .5'»"57 




-MATRIX: Adequacy 
of i ntcrpretat i on 
of learn i ng 
fai 1 ures • , « , 61 

■» 


SUMMARY OF 

PROCEDURES . . . 60 




-MATRIX: Types 
of program 
weakness • * 67, 69 , 
70 


-MATRIX: Revising 

program . . . . 68, 71 
-MATRIX: RevisinQ 

for speci f i c 

di fficul ties . 76-81 
-MATRIX: Priori ties 

to observe vsfhcn 

revi si ng program . 73 


-MATRIX: Adequacy 
of program 
revision .... 87 


SUMMARY OF 

PROCEDURES ... 86 



J. 1.2 



].1.3 



.1.4 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


— A small sample of from tioo to five students will be 
ideyitifiedt They will be representative of the 
target audience but will be at the lower end of the 
profioieiicv distribution with respect to the be^ 
havvor to be taught. 

— Those portions of the criterion behavior already in 
their repetoire will be identified. 


WHAT YOU WILL 
WORK FROM 


(1) The target audience to be taught. C\ 

(2) Sub-Criterion and Criterion tests for the 
material to be taught. 


WHAT YOU WILL 
DO 


(1) Select a sample of students from the target 
audience. 

(2) Administer the tests to them as a pre'" test. 

(3) Identify lov; scorers. 

(k) Select a sample of from two to five of the low 
scorers. 


FORMS YOU WILL 
USE 


None 
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DESCRIPTION OF Sub-STEP J. 1.1 





INPUT 




ACTION 




OUTPUT 




Target audience 




Select representative 




Sample of identified 










sample 




"low" proficiency 










and administer 




+ 








► 


sub-criterion and 




identification of 










criterion tests 


► 


portions of 










to them 




criterion behavior in 






i 




ii 




its repertoire 

^ 111 















Job Aid Contents 



CRITERIA FOR 



IDENTIFYING INPUTS 


ACTION TO BE TAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 




-MATRIX: Informa- 






-MATRIX: Adequacy 


SUMMARY OF 




tion obtained 






of sample selection 


PROCEDURES . . . 


20 


from "before" 






and use of 






and "after" 


16 




pre- instruct ion 






tests 




test results . . 21 






-MATRIX: 












Properties of 












adequate tryout 












sample 


17 











Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLi.TED FORMS 

STEP 


BLANK FORMS 


1 dent i f i cat i on of 
audience characteris- 
tics 


A. 1.2 


Sub-cri ter .on and 
criterion t-sts: 
FORM F.2(l) 
(carried forward frorri) 


1.2 




Identification of 
audience characteris- 
tics (based on tests) 


F.^.2 
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Sub- STEP J. 1.1 



INPUT 



Target audience 



JOB DIAGRAM 



ACTION 



Select representative 
sample 
and administer 
sub-criterion and 
criterion tests 
to them 

ii 



OUTPUT 



Sample of identified 
"low" proficiency 
+ 

identification of 
portions of 
criterion behavior in 
its repertoire 



iii 



Sample which 
is 

REPRESENTATIVE 
of target audience 



Select for 
informal tryout of 
f ns t rue t f ona 1 prog ram 



M .a 



Use of smal 1 sample 
of 2-5 students 
resembl Ing the 
target population in: 

-General abi I ities 

-Specific abi 1 i ties 

-Pre-instructional 
proficiency 

ill .a 



Sample which 

IS 

NON-REPRESENTATIVE 
of target audience 



Reject for 
informal tryout of 
I ns t ruct i ona 1 prog ram 



ii .b 



Non-use of 
sample which 
Is too deviant 
from target population 



iii.b 
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BACKGROUND INFORMATION 





page 


Information obtained from tests 
administered •'before" and "after" 
students receive instruction 


16 


Properties of an adequate sample 
of the target audience for purposes 
of the informal program tryout 


17 
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J. 1.1 

IDENTIFICATION 
MATRIX 



DIFFERENT TYPES OF INFORMATION OBTAINED BY TESTS 
AOMISISTERtD ^'BEFORE'' AND "AFTER" THE INSTRUCTIONAL PROGRAM 



TIME OF 
TEST 
ADMINISTRATION 



Test administered 
BEFORE 
I nstruct i onal program 



CRITERIA 



Test administered 
AFTER 

Instructional program* 



Test results can be used to 
identify what students can do 
after they have taken the 
znstructional program: 



"Test results can be used to 
identify what students can do 
befor e they^have taken the 
instructional progrrm: 

••Whether they cxm exhibit the total criterion behavior 

••Whether they can exhibit one or more sub-criterion behaviors 
which make up the criterion behavior 

••Whether they can exhibit one or more of the component skills 





-Permits the developer to 


-Results on "after'' test, when 




differentiate between satisfactory 


compared with those on "before" 




performance on program problems 


test, can be used to identify 


RATIONALE 


and tasks which is due: 


how well the program builds 


FOR USE 


••To students' entering 


proficiency at: 




proficiency; OR 


••Criterion behavior 




••To the capacity of the program 


• -Sub-cri terion behaviors 




to Instruct 






-If the student can perform before 






beginning instruction, his 






performance on the program reveals 






noth ing about the program ' s 






adequacy 





*Tests administered after In F ormal tryout produce equivocal results. This is so 
because of the interview procedures used during the program tryout and the effects 
that questions may have on what is learned. For this reason, it is probably 
better not to administer the tests after instruction (or if they are used, to 
Interpret their results with caution). "After" tests are used during the 
developmental tryouts (See STEP J. 2). 

16 
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IDENTIFICATION 
MATRIX 



PROPERTIES OF AN ADEQUATELY SELECTED SAMPLE 
OF TARGET AUDIENCE FOR ADMINISTRATION OF PROGRAM (PRIOR TO REVISION) 



PROPERTIES 
OF SAMPLE 



SIZE 



REPRESENTATIVENESS 



NON-PROFICIENCY 
at criterion behavior 



CRITERIA 



'Each try out (before 
revision) should be 
based on a sample of: 

• *A minimum of two 

students 
AND 

• v4 maximum of 

approximately 
five students 



'The sample of students 
selected for informal 
try outs should reflect 
the target audience 
for: 

••General ability 

• •Relevant J special 
abilities 

• •Estimated levels of 
entry proficiency 
(at criterion 
behavior) 



'77te sample of 
students selected for 
informal tryout 
should: 

••Not consist of 
students from the 
upper portion of 
the distribution 
of scores 
identifying 
pre -instructional 
proficiency at the 
criterion behavior 



RATIONALE 



-A sample size of from 
two to five is: 

••Of sufficient size 
to reveal program 
weaknesses 

••Not so large that 
the interview/ 
probing procedures 
used are too time- 
consuming and 
costly 



er|c 



■Adequate accommoda- 
tions of the target 
audience (i.e., 
producing a program 
on which most students 
can take in 
relatively error-free 
fashion) depends on 
the sample represent- 
i ng : 

••The entry behaviors 
likely to influence 
performance 

• •The 1 ikely perfor- 
mance of the target 
audience on the 
program 



17 



"If tryout is 
performed wi th 
students who already 
are highly proficient, 
the tryout wi 1 1 not 
be capable of 
assessing the adequacy 
of the program for 
bringing up to 
proficiency those 
students who are not 
proficient to begin 
with 



JOB PROCEDURES 





page 


SUMMARY OF PROCEDURES 


20 


Adequacy of procedures for selecting 
sample of target audience and for 
use of pre-instruction test results 


21 
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J. 1.1 

STANDARDS 
MATRIX 



ADEQUACY OF SAf-IPLE SELECTION 
(AND USE OF PRE-INSTRUCTION TEST RESULTS) 



PROPERTIES 



REPRESENTATIVENESS 



-Sample selected is 
representative of 
(i.e. J measures at 
approximately the 
CRITERIA I mea^x of distributions) 
the target population 
in: 

"General ability 

' 'Specific abilities 
(relevant to mastery 
of the criterion 
behavior) 

• *Pre-instructional 
proficiency in the 
criterion behavior 



SIZE 



-Sample for each 
tryoiit and revision 
cycle ranges from: 

• *Tix> students 
to 

' 'Approximately 
four or six 



DEGREE OF 
PROFICIENCY 



'Sample has relatively 
low proficiency in 
criterion behavior 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


You will have identified: 

— The program problems on lohich errors have been made. 
"^The type(s) of learning failure identified by these 
errors. 

""The reasons or sources of those failures. 


WHAT YOU WILL 
WORK FROM 


(1) Sample of two to five students 

(2) The program to be taken by them 


WHAT YOU WILL 
DO 


(1) Administer the program to one student at a time 

(2) Hake a record of errors which ^occur 

(3) Probe for the type of error and its cause 


FORMS YOU WILL 
USE 


FORM J. 2(1) (Optional) for recording errors on the 
program. 
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DESCRIPTION OF Sub-STEP 



J. 1.2 



INPUT 



of students 
selected for 
informal tryout 



ACTION 



Administer 
instructional program: 

-Observe and make a 
record of errors, AND 

-Probe for type and 
source of failure 



OUTPUT 



Identification of: 

-Program problems on 
which errors were made 

-Type of learning 
failxire 

-Sources of the 
failiires vi 



Job Aid Contents 

CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: What 
information 
observation 
provides . . . 

-MATRIX: When 
probes are 
needed . . . . 



31 
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•MATRIX: 
Priorities 
types of probes 
to use 35 

•MATRIX: Types 
of probes 

to use .... 36-^2 
•MATRIX: Timing 
the use of 

probes 'i3 



-MATRIX: Adequacy 
of observational 
and probing 
process i\j 



(Optional ) 
FORM J. 2(1) 

SUMMARY OF 
PROCEDURES , 



^6 



Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


1 

BLANK FORMS 


Selection of sample 


J.I.I 






FORM J. 2(1) ~ Optional 


1 ns t rue t i ona 1 p rog ram 


1.2 














































1 
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Sub- STEP { J. 1.2 [ 



JOB DIAGRAM 



INPUT 




ACTION 


OUTPUT 




Sample of students 
selected for 
informal tryout 


► 


Administer 
instructional program: 

-Observe and make a 
record of errors , AND 

-Probe for type and 
source of failure 

V 




Identification of: 

-Program problems on 
which errors were made 

-Type of learning 
failures 

-Sources of the failures 

vi 






iv 

















NO error made 
on program problem 



IV. a 



Continue administration 
of program 



Completed program 



VI .a 



Group I 

Zfror made on 
program problem 



iv. 



-Keep a record 

-Continue administra- 
tion 

-Probe for error 
sources of program 
administration 



^b 



■Record of which 
program problems 
produce errors-^- 
uncontaminated by 
probing questions 

-Probing results 



VI .b 



Group i i 

Error made on 
program problem 



I v.c 



■Keep a record 

-Probe for source of 
error immediately 

-Continue probing 
unti 1 source is 
ident I fled 



v.c 



-Probing results 
ident i fy Ing ; 

••Type of learning 
f a i I u re 

••Sources of failure 

••Program features to 
account for failure 



VI .c 
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BACKGROUND INFORMATION 





page 


Nature of informal tryout varies 
depending on type of media used in 
administering an instructional program 


26 


Determining how to conduct an 
Informal tryout for programs 
involving different types of media 


27 


REVIEW SUB-STEP F.3.2 FOR TREATMENT 
OF "PROBES" USED FOR DIAGNOSING 
SOURCES OF ERRORS MADE ON CRITERION 
TESTS 


VOLUME 
F 

96-117 


FORM J.2(l), SUMMARY OF ERRORS ON 
CRITERION TEST, also available on an 
optional basis for summarizing errors 
on programs 


28, 29 


Two sources of information used in 
informal tryouts: observation and 
probi ng 


30 


What you wish to find out by 
"observation" 


31 


What you wish to find out by 
"interview probes" 


32 


Procedures for "probing" 


33 
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J. 1.2 

IDEt4TIFICATI0N 
MATRIX 



MEDIA CALLING FOR DIFFERENT TRYOUT APPROACHES 



TYPES OF 

urn 1 A 


Media It Is 1 ikely to be 
economtcally or loglstlcally 

r cMo 1 DLL 

to use during Informal tryout 


Media it Is likely to be 
economically or logistical ly 

to use during Informal tryout 




p.— — — „■.,„■■.■■■■■■■■■■■■,. — 

-PRINT meava 






• * Programs requii^ing paper-and- 
pencil reaponding 






-ENVIRONMENTAL media 


-'ENVIRONMENTAL media 


CRITERIA 


* •Objects or equipment already 
available and for which the 
risk or cost of damage is 
slight 


••Objects or equipment not yet 
on hand and either costly to 
acquire or involving high 
damage risks 




••Already built simulators 


••People to whom or for whom 
there is a risk factor 




-DEVICE media 


--DEVICE media 




••Those media which prove not 
costly hut only because the 
extent of their use is law 
for the instructional program 
at hand: 

/Slides 
/Audio tape 


••Those media for which production 
costs are high: 

/Film, animation 
/TV 

••Those media wl^ose use will be 
Qostlu because of the extensive 
use required by the instruationd 
program at hand: 

/Slides 
/Audio tape 
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J^U2 
DECISION 
MATRIX 



DETERMINING HOW TO CONDUCT INFORMAL TRYOUT 
FOR PROGRAMS INVOLVING DIFFERENT TYPES OF MEDIA 



CONDITIONS 



Economically and logistical ly 
FEASIBLE 
to use medium 
required by the 
instructional program 



ACTION 
TO TAKE 



'Use the actual media which ajce 
catted for by the instructionat 
program in the i nformal try out 



Economically and logistical ly 
UNFEASIBLE 
to use medium 
required by the 
instructional program 



-Conduct informal try out 8 with 
storyboard versions of 
instructionat program: 

* •Paper-and-pencit scripts 
replace dialogue (to appear 
later on film^ TV^ or tape) 

• •Drawings or live demonstrations 
to replace visual events 

••Mock-ups to replace actual 
objects 

••Actors to replace the people 
who are usually found in the 
criterion situation 

-Conduct developmental try outs 
(See STEP J, 2) with actual media 
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* 


-FORM J. 2(1) was designed to serve in 


summarizing errors 


on criterion tests (See STEP J. 2). 




-It is also dVdilab!e» on an optional 


basis, to serve in 


summarizing errors which students make on the program 


during its informal tryout or during 


i ts devel opmental 


tryout . 





HOW TO USE THE FORM 

(See opposite page for an example of a filled out FORM) 

-A column is used to record tallies for the number of 
students making errors on each numbered problem or task. 

-Each row (labeled a-e) may be used to record the follow- 
ing types of information: 

••For multiple choice type problems, which i ncorrect 
options were chosen. 

••For problems where more than one response is required, 
which of the responses \s incorrect 

-Two rows (with black border) are available for summariz- 
ing the total number of people making errors on a program 
problem and that number expressed as a percentage of the 
total number of people taking the program* 

-The bottom section of the FORM can be used to characterize 
the nature of the performance required on the problem; 
i.e., RECALL, because It Is a problem that repeats an 
INPUT or OUTPUT, or TRANSFER, because the problem involves 
new INPUTS or ACTIONS.. 

-Sub-criterion problems or criterion problems can be 
identified by putting one or more asterisks next to the 
number corresponding to them. 
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(SUB ) CRI TERION BEHA VIOR \ \ SUMMARY OF ERRORS ON CRITERION TEST 



ITEM #'S Q 




2 


3 


4 


5 


6 


7 


8 


* 
9 


10 


11 


12 


13 


14 


16 


16 


17 


IB 


19 


20 


21 


22 


23 


24 


26 


•. 


// 




/ 












/ 


































b 






/ 




// 
























r 














c. 


/ 












/// 


































d 


// 


















































«. 




















































Toul Number 

of Pwpis 
Making Errort 


5 


0 


2 


0 


2 


0 


3 


0 


I 


































% of Group 
Miktng Erron 


100 


0 




0 


ko 


0 


60 


0 


20 


































Raquirtmtnu 
^ 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 


16 


17 


18 


19 


20 


21 


22 


23 


24 


26 


INPUT recall 


X 






X 


X 




X 






































INPUT ir«niftr 




X 


X 






X 




X 


X 


































AgfTION recall 


X 


X 


X 


X 






X 






































ACTION trtntf*r 










X 


X 




X 


X 



































^ub-crl tarlon problem 
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J. 1.2 
IDENTIFICATION 
MATRIX 



TWO SOURCES OF INFORMATION ABOUT PROGRAM ADEQUACY 
WHICH ARE USED IN INFORMAL TRYOUTS 



SOURCES OF 
INFORMATION 



OBSERVATION 



CRITERIA 



-Ab an individual student works 
on each program problem or task 
(presented by prints environ- 
mental^ or device media) ^ the 
developer observes student 
perfomance in order: 

• •To determine whether the 
student has made an error on 
that problem; 

Am 

••To surmarize the error results 
for the whole succession of 
program problems leading up 
to the criterion px^blemCs) 



INTERVIEW PROBES 



-Wten it is detemined from 
observation that a student has 
made an error on a particular 
problem^ he is asked probing 
questions in order: 

••To identify the source of or 
the reason for student failure 
to get a correct answer 
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0.1.2 
IDENTIFICATION 
MATRIX 



TYPES OF INFORMATION YOU WISH 
TO OBTAIN FROM ^^OBSERVATION'' 



INFORMATION 
SOUGHT 



CRITERIA 



On WHICH problems 
are errors being made? 



WHAT TYPE 
of errors 
are being made? 



-On which (numbered) 
problems are errors 
made? 

-On multiple choice 
problems^ which 
incorrect options 
are endorsed?* 

-On a problem with 
more than one part 
to it ^ on which parts 
are errors made? 

-Is there a succession 
of problems on all of 
which errors are made? 

-Where in a succession 
of problems is an 
isolated error made? 



'Are errors made on: 

••Criterion problems 

• •Sub-criterion 
problems 
OR 

••'^Frogression^^ 
problems 

--Are errors being made 
on problems that 
involve: 

• •RECALL 

AND/OR 
••TRANSFER 

-Are errors being made 
on problems that 
involve only one 
component skill (and 
whose results are 
therefore diagnostic 
of a specific 
problem)? 

-Is an error one of 
omission (i.e.j no 
answer given) or of 
commission (i.e.j 
wrong answer given)? 



HOW MANY people 

are making 
HOW MANY errors? 



-How many people 
(what percentage of 
all those doing a 
problem) make errors 
on a problem? 

"On multiple choice 
problems^ how many 
people endorse which 
wrong options? 

-On multiple part 
problems J how many 
people make errors on 
particular parts? 

"On how many problems 
in a series leading 
up to a criterion or 
suL -criterion are 
thei e errors ? 



^Whenever a particular option Is associated with a particular type of error 
(e.g., association or discrimination), these results can be diagnostic. 
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0.1.2 

IDENTIFICATION 
MATRIX 



TYPES OF INFORMATION YOii WISH TO OBTAIN FROM "PROBES" 



"T 



!NFORMATION 
SOUGHT 



WHICH COMPONENT SKILL 
does the foi lure 
induce? 



CRITERIA 



-Which component skills 
does the failure 
involve : 

• •Discriminations 

• •Generalizations 

• •Associations 

OR 

• •Chains 

^Does the failure 
consist of: 

• •The skill being 
incorrectly learned 

OR 

• •The skill not 

having been learned 
at all 



WHICH PROPERTIES 
of INPUTS, ACTIONS. 
OUTPUTS 
account for 
the fai 1 ure? 



-yhich properties of 
INPUTS, ACTIONS, or 
OUTPUTS: 

••Do students overlook 
or not pay attention 
to 

OR 

••Do students confuse 

^Are there properties 
of the examples of 
INPUTS, ACTIONS, or 
OUTPUTS which are 
used which result in: 

•'Student confusion 



DOES PROBLEM FORMAT 
account for 
the fai lure? 



-Is the error due to: 

• 'The absence of 
inadequate or 
confusing instruct- 
tions about the 
problem task 

• •A vocabulary or 
readability level 
inappi jpriate for 
the target audience 

• • The absence of 

or presence of 
inadequate or 
confusina 

techniques used to 
direct attention or 
'observation 
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THE "PROBING" PROCESS 





page 


When probes are needed 


3^ 


Priorities in the types of probes 
used 


35 


Some general probes to use to 
identify problems 


36 


Follow-up probes to use in response 
to student answers to initial 
probes 


37, 38 


General vs. specific types of 
probes 


39 


Responses to general probes which 
do not unambiguously identify the 
type of learning failure 


^0 


Specific probes which can be used 
to Identify the type of learning 
fai lure 




Why probes are not needed concern- 
ing "chaining" failures 


m 


Timing of probes 


^3 
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J. 1.2 

DECISION 
MATRIX 



DETERMINING WHEN PROBES ARE NEEDED 





An error Is made 


An error is made 




on a problem or task 


on a problem or task 


CONDITIONS 


AND 


AND 




The problem or task 


The problem or task 




involves MORE THAN ONE 


involves ONLY ONE 




component skM 1 


component skM 1 



ACTION 
TO TAKE 



'Ppobe in order to find out 
which component skill mcq^ not 
have been learned* ^ 



"Do not probe; failure on the 
problem identifies which 
component skill: 

• *HajB not been learned 
OR 

••Has been learned incovrectly 



*The same wrong response may be 
Indicative of a failure to learn 
a discrimination, generalization, 
or association 



EXAMPLE 



Example of a non- d iagnosti c problem 

Tht man wadfe to mhk. 

Tht man (^alfu to loofik. 



An error could be due either to a 
failure to discriminate between 
singulars or plurals; or to a 
failure to associate the proper 
form of the verb wi th singular and 
pi ural forms. 



Example of a diagnostic problem 
"Both 4el^^^nc^6 fae^ow; uaz a 

WkLck had thz con/izct veAb?" 

Thz man woife to wofik. 

T ke. man u)atk6_ to uooKk. 

An error Is likely to be due to 
a failure to associ ate the proper 
form of the verb with a singular 
subject. (This assumes that the 
student has already learned what 
a singular subject is.) 
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DETERMINING PRIORITIES IN THE ORDER IN WHICH 
DIFFERENT TYPES OF PROBING QUESTIONS SHOULD BE ASKED 



PRIORITIES 


FIRST 


If f i rst answers 
provldf; Indeterminate 
results 

SECOND 


If second answers still 
provide Indeterminate 
resu] ts 

THIRD 


ACTION 
TO TAKE 


-Ask open- ended 
questions which do not 
identify or suggest 
answers student might 
give 

-Do not call attention 
either to the IIWDTS 
or to the ACTIONS for 
him to focus on 

—Do Yiot cal I attention 
to specific properties 
of INPUTS or of 
ACTIONS to focus on 

-Do not direct 
attention to any 
features of prob lem 
1 format 


-Ask directed questions 
which identify what 
the student should 
constder when trying 
to assess what went 
wrong 

-Call attention 
generally either to 
the INPUTS or to the 
ACTIONS for him to 
focus on 

OR 

-Call attention 
generally to the 
problem format j BUT 

—Do Hot cal I attention 
to specific properties 
of INPUTS or of 
ACTIONS to focus on 

-Do not call attention 
to specific properties 
of the problem format 


"Ask directed questions 

"Call attention to 
INPUTS or to ACTIONS 

-Call attention to 
specifia properties 
for student to focus 
on 





-"Why did you 

?" 


-"What is there about 
this that made 


-"Which of these 

of the 


EXAMPLES 


take this ACTION 


INPUT 

you ?" 


properties 

made you 




take this ACTION 


INPUTS 

?" 




e.g., "Why did you 


e.g., "What pKopznty 
oft aWUhiitz 0|J 
tSoAJi Lza{^ made, 
tjfoa cZcL!>6'Lfiy It 
tkl& my? 


take this ACTION 
e.g., "Wkidi oi thdJdz 

hAjzz, Afiape, oh. 
cotoK made, you 
make, thi6 
poAttcuiZa/i 
cZa^6lfilcation?" 




-'Ms there anything 
about the way the 
problem is presented 
which led you to give 
your answer?" 


-"Did you have trouble 
wi th : 

• •T/ie voaibuZaA,ij 
••T/ie iyi&tAactiond 
••T/ie mzaning oi 

Aen;tence6 
••T/ie a6e tko, 

diagram 
'-Etc. 


-"Did you have trouble 

wi th : 

• • TftcA dilag^am 
**Thl& lUjuUtt/iaXion 

• •The, 4^Ku.|$-ccance OjJ 
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DECISION 
MATRIX 



DETERMINING WHAT GENERAL PROBES TO USE 
(ON PROGRAM PROBLEMS WHICH ARE NON-DIAGNOSTIC) 



CONDITIONS 



" NO answer is given 
- NO procedure or ACTION Is taken 
on program problem 



ACTION 
TO TAKE 
(by priorities) 



#1: OPEN-ENDED PROBE 

'What's giving you trouble on 
this problem? 

'*What did yc/u leave out the 
ansioer (fail to do something)? 



#2: LESS OPEN-ENDED PROBE 

'Are there any answers you 
thought of but ruled out? (If 
80j why?) 

'Are there any changes that could 
be made in the problem that 
would have enabled you to solve 
it? 



#3: MORE DIRECT PROBE 

mjf r ^^It "'Qll^ ^4.4.^ 



of the 



tyfl UJ unlets 

or 



property INPUT ACTION 

can you give the answer now? 

'Hai^e aX'e some possible answers; 
which is the right one? 



-A WRONG answer is given 

-A WRONG procedure or ACTION is 
taken 



#1: OPEN-ENDED PROBE 

•What made you give the answer 
you didfi 

'What is there about the problem 
that led you to give that 
answer? 



if 2: LESS OPEN-ENDED PROBE 
« 

'Are there any other answers you 
thought of but ruled out? (If 
so J why?) 

'What changes in the problem 
would lead you to give another 
answer? 



#3.- MORE DIRECT PROBE 

-If I call your attention to this 

of the or ^ 

property INPUT ACTION 

would you still give the same 
answer? 

'Which of these properties j 

^ _p or J did you 

consider relevant ( irrelevant) ? 
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DETERMINING WHAT KINDS OF FOLLOW-UP PROBES TO USE 
IN RESPONSE TO STUDENT ANSWERS TO INITIAL PROBES 
(ERROR ON PROGRAM: NO ANSWER) 



CONDITIONS 


-Type of error on 
program: 

• -No answer 
•f 

-Response to earlier, 
more open-ended probe: 

••^M DON'T KNOW'' 


-Type of arror on 
program: 

• -No^ answer 
+ 

-Response to initial 
probes: 

••Cites possible 
answers or proper- 
ties of INPUTS or 
ACTIONS about whi ch 
he is UNCERTAIN 


-Type of error on 
program: 

• •No answer 
+ 

-Response to initial 
probes : 

••Cites irrelevant or 
incorrect properties 
of INPUTS or ACTIONS 

• •Cites incorrect 
answers 


ACTION 
TO TAKE 

- 


-Use less open-ended 
probes 

-Continue using 
increasingly more 
direct probes until 
student identifies th'd 
source of the 
diffioulvy: 

• • Features of the 
problem content 
OR 

• •Features of the 

problem format 


-Probe for which answer 
he thinks correct 
(and why) 

-Probe for reason for 
his uncertaintij 

-Probe for student 
idea of the relation- 
ship between 
properties he 
identifies and 
possible answers 


-Probe for why he 
cited these 
properties and how 
they relate to the 

answer 

-Probe for why he gave 
the answers he did 

-Probe for alternative 
answers he considered 
and why he rejected 
them 



EXAMPLES 




SEE PAGE 3b 



ijoa mtntiomd do vfotx 
think 4J> cow^cX? 

-"What U It that 
mafeeA ijovL unc^Atciin?'^ 

-"l^hat QJUz do ijovL (^tol 
tjou need to fenow) in 
oKdoA to decxdef" 
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did uoa ^avj 

incon/izct an6tOe/T, 
-*'Wht^ did tWVi 6aij 

incon/izct/ViAtlevant 
711 



p^opt/itfj 

'"Did (fou coMidQA ancf 
otiieA 



pAopeAtyte^ 



J. 1.2 

DECISION 
MATRIX 



DETERMINING WHA-T KINDS OF FOLLOW-UP PROBES TO USE 
IN RESPONSE TO STUDENT ANSWERS TO INITIAL PROBES 
(ERROR ON PROGRAM: INCORRECT ANSWER) 





-Type of error on program: 


-Type of error on program: 


CONDITIONS 


••Incorrect answer 
+ 


• • Incorrect answer 




-Responses to earlier, more 
open-ended probe: 

••"1 DON*T KNOW" 


-Responses to initial probes : 

••Cites irrelevant or Incorrect 
properties 


ACTION 
TO TAKE 


"Uae less open-tended probes 

'Continue using increasingly more 
direct probes until student 
identifies the source of the 
difficulty: 


1 

'Probe for why he cited these 
properties and how they relate 
to the answer 

'Probe Tor whu h.p onvti thp nyiswey^s 
he did 




* ^Features of the problem content 
OR 

Features of the problem format 


'Probe for alteiviative answers he 
considered and why hfc> rejected 
them 







-^*iji}hif did ifOii 6aif 








EXAMPLES 


SEE PAGE 3b 








-"W/iy did ijoii 4iatj 


















- **Vld ifoa con&ido.K any othoA 






?" 
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IDENTIFiCATiON 
MATRIX 



PURPOSE OF TWO TYPES OF PROBES: 
GENERAL VS, SPECfFIC 



TYPES 
OF PROBES 



GENERAL 
PROBES 



SPECIFIC 
PROBES 



The ultimate aim of both types 
of probes is to identify the 
specific type of teaming faitiwe 
that occTred (i,e.^ discrimina- 
tions^ generalizations y 
as sedations J or chains) and the 
reasons for the failure. 



'The probing process begins with 
general y open-ended probes in 
order not to provide the student 
with clues as to possible 
responses to the pi*obea 



--Following responses to general 
probes^ if it is still not clear 
which type of learning failure 
has occurred and why it has 
occurred^ probes specifically 
designed to identify the failure 
are used 



EXAMPLES 



SEE PAGES 
36-38 



SEE PAGES 
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IDENTIFICATION 
MATRIX 



SITUATIONS FOLLOWING RESPONSES TO GENERAL PROBES 
IN WHICH IT IS UNCLEAR WHICH COMPONENT SKILL(S) HAS OR 
HAS NOT BEEN IDENTIFIED AS THE ONE NOT LEARNED 



SITUATIONS 
FOLLOWING 
USE OF 
GENERAL PROBES 



Uncertainty whether 
there has been a 
failure to learn 

DISCRIMINATIONS 



Uncertainty whether 
there has oeen a 
failure to learn 

GENERALIZATIONS 



Uncertainty whether 
there has been a 
fa\ 1 ure to learn 

ASSOCIATIONS 



CRITERIA 



Reeponses to 
general probes haoe 
failed to indicate: 

'Whether a student can 
or cannot tell the 
difference between 
INPUTS belonging to 
different classes 
because he does none 
of the following; he 
does NOT: 

• • Identify (correc t ly 
or incorrectly) the 
class to which the 
example used in the 
problem belongs 

• •Attribute class 
properties (correct^ 
ly or incorrectly) 
to the example 



Responses to 
general probes have 
failed to indicate: 

-Whether a student can 
or cannot see the 
similarity among 
INPUTS belonging to 
the same classes 
because he does none 
of the follntoing; he 
does NOT: 

• •Identify ( correct ly 
or incorrectly) the 
class to which the 
example used in the 
problem belongs 

••Attribute class 
properties ( correct^ 
ly or incorrectly) 
to the example 



Responses to 
general probes hcxoe 
failed to indicate: 

'Whether a student can 
or cannot tell which 
ACTION QO^s mth 
which INPUT because 
he does none of the 
follobrLng; he does 
NOT: 

• •Identify all 
possible ACTIONS 
that could be taken 
(depending on which 
class of INPUTS is 
involved) 

• • Identify all 
possible INPUTS 
which would require 
different actions 

••Mismatch INPUTS 
and ACTIONS 



EXAMPLES 



ERLC 



e.g., 

-The 6tadtnt koA on a pn^oQKom pfiobtm 
inconAzctly labztad A Izve/i (U a 2nd c£a44 

-In leAponAe to pKobt6 fie doe^ not 6cuf u)hy he 
taboJtzd it that u)ay; he doesn't 6ay, ion. 
examplz, becooAe the z^onX ion.cz bztwzzn 
the n.^^>Utancz ({o^ce ayid thz {^lOLz/tm 



e.g. , 

-The 6tudznt do^n^t 
idzYvti(^ij pn.op^Atit6 

0({ tilt thAdZ ct<lSi6PJi 

oi £eve^6 and thzn 
indyicatz that he 
doesn't knoiA) lohtck t& 
cjJiiJidd Ut, 2nd, on. 
5n.d a£a6<6 
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J. 1.2 

DECISION 
MATRIX 



DETERMINING HOW TO PROBE SPECIFICALLY 
FOR SPECIFIC TYPES OF LEARN SNG FAILURES 
WHEN GENERAL PROBES HAVE NOT IDENTIFIED THEM 



CONDITIONS 



ACTION 
TO TAKE 



Uncertainty whether 
there is a 
DISCRIMINATION 
failure 



'Piesent additional 
examples from a 
different INPUT alass 
which is represented 
in the original pro^ 
gram problem and ask: 

^•Are these the same 
or different? 
OH 

••Should these be 
handled or treated 
the same way or in 
different ways? 

-If examples vary 
dimensional ly^ present 
pairs of INPUT 
examples (using the 
original example as 
one member) which 
start far apart and 
gradually are made 
more similar 

• •Deteimne when and 
where discrimination 
breaks down 



Uncertainty whether 
there is a 
GENERALIZATION 
f al lure 



'Present additional 
examples from the same 
INPUT class which is 
represented in the 
original program 
problem and ask: 

• *Are these the same 

or different? 
OH 

••Should these be 
handled or treated 
the same way or in 
different ways? 

^If examples vary 
dimensi ona I ly^ present 
pairs of INPUT 
examples (using the 
original example as 
one membeiO which 
start close together 
and gradually are 
made more dissimilar 

• •Determine when and 
where generalizatim 
breaks down 



Uncertainty whether 
there is an 
ASSOCIATION 
fal lure 



"If behavior has been 
taught in two 
directions : 

INPUT ACTION and 
ACTION INPUT: 

•* Reverse a test 
question and 
determine whether 
same mismatch 
occurs 



'Identify for the 
student all the 
classes of inputs and 
all the actions triat 
can be taken and then 
ask: 



•Which 

ACTION 
with which 



goes 



INPUT 



•This is how I pair 
them up. Am I 
right or wrong? 



EXAMPLES 



anothoA Zzvtn. {{^Kom 4ame on. diiffOAzrvt 
otiOA^); X6 tiuJ} likt tkt one. In tht pKobtm 
on. dyi^^eAznt ^n.om itV^ 

"What do ihet^f have. In tommon?^^ 

OR 

"In wfiat way o/ie they cU^^^ent?" 



e.g. > 

"He/ie (Viz zxampl2J> 

IzvQM. Which otJ thorn 
bztong^ to tkt l4>ty 
to thz 2nd, and to thz 
3n.d cla66?'' 
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IDENTIFICATION 
MATRIX 



DIFFERENCES IN NEED TO PROBE FOR COMPONENT SKILLS 



PROBING 
NEEDS 



CRITERIA 



There IS a need to probe 
for failure In: 

-Discriminations 

-General Izatlons 

-Associations 



-i4 urong answer for a single 
Sub-STEP (involving just one 
assocrLation) can be due to one 
or more of the above types of 
fai lure 

-ThereforBy probing is designed 
to find out which of the types 
of failures ocaurred 



There Is NO need to probe 
for fal lure In: 

-Chaining 



-Probing is used to find out the 
type of failure involved within 
each of the separate Sub-STEPS 
in a chain 

-When the problem task involves 
producing a series of Sub -STEPS 
(i^e.y a chain) y the perfomanoe 
itself reveals whethsr: 

••i4 Sub-STEP is omitted 

••Sub-STEPS are performed in the 
wrong sequence 

-Probing is therefore unnecessary 
to identify a ''chaining failure'^ 



*Probing for student reasons for omissions or incorrect sequences can be used. 
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DECISION 
MATRIX 



DETERMINING WHEN TO PROBE AND HOW TO TIME "PROBING" 



GOALS 



For obtaining OBSERVATIONAL 
error data 
uninfluenced or uncontami nated 
by probing procedures 



For obtaining 
responses to PROBES 



ACT 1 01^ 
TO TAKE 



'Administer the entire program to 
one or two students of the sample 



'Observe occurrences of errors— 
but do NOT probe for nature or 
causes of problem at the time 



-Allob) student to complete the 
program and then go back to 
program problems or tasks and 
begin probing 



'See Way #1 in the left-hand 
column 

•f 

'Administer the entire program 
to one or two students in the 
sample 

'Observe occurrences of errors and 

• ' Before the student is given 
feedback 

• * Begin probing 

'Use general probes before using 
specific probes 

'When using specific probes^ 
probe for failures in this order: 

• 'Re: Discriminations 

• Generalizations 

• •i?e: Associations 



RA.TiONALE 



ERJC 



-Probing while the student Is 
taking the problem will disallow 
an uncontami nated assessment of 
the capacity of the program 
sequence (as originally developed) 
to bring students up to 
prof i ciency 

-Probing Immediately after an 
error provides clues to the 
student which may help him on 
subsequent problems 



-Probing before feedback !S given 
allows determination of whether 
the student has learned anything 
(unprompted by the feedback) 



1*3 



JOB PROCEDURES 





page 


SUMMARY OF PROCEDURES 




Adequacy of "probing for 
sources of learning failure" 





















J. 1.2 



ILLUSTRATION SUMMARfZING PROCEDURES INVOLVED IN 
ADMINISTERING PROGRAM, RECORDING ERRORS, AND 
PROBING FOR SOURCE OF LEARNING FAILURE 



#1 



n 



LOOK FOR ERRORS 

a. Administer Instructional 
program I ndlvidual ly 

to each student In the 
sample 

b. Observe occurrence of 
error (s) on program 
problem or task 

c. Make a record of 
occurrence and of the 
type of error* 

d. (OPTIONAL) Use FORM J. 2(1) 
to summarize errors on 

all program problems 



c. 



PROBE FOR REASONS 
FOR ERRORS 

Use general probes for 
identifying source of 
errors: 

••Start with open-ended 
probes ; 

••Continue with less 
open-ended probes ; 

• •Use di rect probes 

Use probes to follow up 
student responses to 
initial probes 

Use specific probes to 
identify specific problems 



*0n printed programs a record can be kept on a copy of the program itself. 

FORM J. 2(1) 
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ITEM «'8 Ij 
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0J.2 

STANDARDS 
MATRIX 



ASSESSING THE ADEQUACY OF THE OBSERVATIONAL AND PROBING PROCESS 



PROPERTIES 



AVAILABILITY OF 
A RECORD 



-i4 record identifying: 

•*All progvm problems or tasks 
on which errors haoe been made 

^*The parts of the problem on 
CRITERIA I which errora have been made 

(ITJiere applicable) the 
incorrect options selected 

A record of responses to probes: 

•'On a printed program itself^ or 

•'0i^ paper for non-printed 
progran's 



COMPLETENESS 



'An attempt to identify: 

'*The nature of the learning 
failure (which skills not 
learned) 

'•The sources of the failure 
(properties of INPUTS or 
ACTIONS causir^ problems) 

'An attempt to identify features 
of the program which can account 
for the failure: 

Content (e.g*j examples used) 

• • Format (vocabu lary^ 
instructions J diagrams^ etc*) 
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PREVtEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


An identifiaat-Can of the nature of the learning 
\ faClur :s which have oocurred for the too to five 
1 students who took the program ^ 


WHAT YOU WILL 
WORK FROM. 


il) A record of student performance (errors) on the 
program. 

(2)' Students* responses to d'fagnostfc probes con- 
cernlna the errors thpv nadp. 


WHAT YOU WILL 
DO 


Identify the types of Teaming failures which have 
occurred and their sources. "* 


FORMS YOU WILL 
USE 


Hone 
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DESCRIPTION OF Sub-STEP 



J. 1.3 





INPUT 




ACTION 




OUTPUT 




Student performance 
on program 
and his^ responses 
to diagnostic probes 


^ 


Identify the 
type of learning failure 
which has occurred 




Identification of types 
of learning failures 
which have occurred 










— ^ ► 








vii 




viii 




ix 















CRITERIA FOR 
IDENTIFYING INPUTS 



Job Aid Contents 



ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



-MATRIX: Types 

of student 

difficulties . . 52 
-MATRIX: Symptoms 

of d i f f i cul t ies 

with program 

format 53 

-MATRIX: Symptoms 

of d i f f i cul t i es 

with program 

content . . . 5^*57 



-MATRIX: Adequacy 
of interpretation 
of learning 
fai lures .... 



FORMS TO USE 



61 



SUMMARY OF 
PROCEDURES 



. 60 



Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Administration of 
program and 
intervif^w rp<;ijll-«; 


J. 1.2 


















































* 









ERIC 



h9 



Sub -STEP I J."!. 3 



JOB DIAGRAM 



INPUT 



student performance 
on program 
and his responses 
to diagnostic probes 



vii 



ACTION 



Identify the 
tjrpe of learning failure 
which has occurred 



viii 



OUTPUT 



Identification of types 
of learninp; failures 
which have occurred 



ix 



STUDENT 








IDENTIFICATION 




-Asks what he is 
Supposed to do 

^Does something other 
than what told to do 




Interpret as 

a format 
di f f i cu 1 ty 





Difficulty with: 
-Vocabu lary 
-Sentence structure 
- Inst ruct ions 
-Di rect ions 






-Asks meaning cf word 

-Pays attention to 
wrong features of task 


— . ^ 








vi i .a 




vi i i .a 




















-On program problems 
deal 1 ng only wi th 
spec i f 1 c component 
skills, commits an 
error 




Interpret as 
a content 
d if f icul ty 

vi i i .b 


► 


Difficulty with: 
-Discriminations 
-General i zat ions 
-Associations 






-In response to probes, 
incorrectly classifies 
INPUTS or incorrectly 
att r i butes proper t les 
to INPUTS ^ 


^ 




-Chai ns 
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BACKGROUND INFORMATION 





page 


Two major types of problems 
students may exhibit: 
format or content 


52 


Evidence indicating format 
diff icul ties 


53 


Evidence indicating content 
difficulties re: Discriminations 


5^ 


Evidence Indicating content 
difficulties re: Generalizations 


55 


Evidence indicating content 
difficulties re: Associations 


56 


Evidence indicating content 
difficulties re: Chains 


57 
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J. 1.3 

IDENTIFICATION 
MATRIX 



TWO MAJOR TYPES OF DIFFICULTIES 
THE STUDENT MAY HAVE WITH A PROGRAM PROBLEM 



TYPES 


1 

CONTENT 


FORMAT 


OF PROBLEMS 


difficulties 


difficulties 



CRITERIA 



'The student has difficulties 
making or forming required: 

• • Discriminations y 
• • Generalizations y 
• •Associations J 

OH 

• •Chains 

'The stu^^nt deals inadequately 
with properties of INPUTS ^ 
ACTIONS, or OUTPUTS: 

••Ignores relevant properties 

••Attends to irrelevant 
properties 

• •Confuses properties 



"The student has difficulties 
with the format of program 
problem or task: 

Fails to understand 
OR 

• * Misunderstands 
OR 

• •Ignores: 

/Vocabulary 

/Task instructions 

/Cues designed to control 
attention or observing 
behavior 

/Statement of objectives 
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EVIDENCE INDICATING WHAT FORMAT DIFFICULTIES 
A STUDENT MAY BE HAVING WITH A PROGRAM PROBLEM 



SOURCE 
OF EVIDENCE 


OBSERVATION 
as a source of evidence about 
format problem difficulties 


RESPONSES TO PROBING 
as a source of evidence about 
format problem difficulties 


CRITERIA 


student failure to understcmd 
instructions or a misunderstandinQ 
of instruction is indicated when 
the student: 

• *Asks what he is supposed to do 

• •Does a task other than that 
called for by instructions 

• *Pay8 attention to features or 

elements of the problem task 
other than those problem 
instructions directed him to 
attend to 

-A student failure to understand 
vocabulary is indicated when: 

• M student asks the meaning of 
a word 


Student indicates in response 
to probes that he failed to 
understand or that he 
misunders tood: 

**What ta^k he was supposed to 
perform on problem 

**The vocabulary used in 
instructions or in the 
statement of problem or in the 
statement of objectives 



e.g., 

-T/ie 6tade,nt coAAyizd a nmeAAxial 
EXAMPLES an&tve/L oat to one dzcAjnal ptacz, 

havtng been triM/iact^d to ca/uiy 
out to /Mee p£ace& 

'*'Ukat ddu '6haptng' tman?*' 



-"I thought I a;a6 6appo^cd to 
coMy thz an^we/t out to one 
decimal p£ace/' 

-"I dtdn^t kymv {A)hat 'Ahaptng* 
meanA • " 
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IDENTIFICATION 
MATRIX 



EVIDENCE SYMPTOMATIC OF CONTENT DIFFICULTIES; 
DISCRIMINATIONS* 



SOURCE 
OF EVIDENCE 



OBSERVATION 
as a source of evidence about 
content dl ff Icul ties 



CRITERIA 



-Commits an error on a program 
problem or txsk which involves 
only ^'disarimirtation" (e,g,j a 
oame/different type problem 
involving INPUTS from different 
classes) * 

-On a given problem^ INPUTS from 
two or more different classes are 
presented; and the student takes 
the same ACTION in response to 
them both (all) 



RESPONSES TO PROBING 
as a source of evidence about 
con tent d i f f i cu 1 1 i es 



IN RESPONSE TO GENERAL PROBES 

-Attributes membership for the 
example of an INPUT (which was 
used in the example) to the wrong 
class 

-Attributes properties to an 
example which it does not have 
(i,e.j they ax*e properties of 
another class) 

-Says he doesn't know which class 
the example belongs to 

IN RESPONSE TO SPECIFIC PROBES 

-WJien presented additional INPUT 
examples (from same or different 
classes) and asked are they the 
same or different j fails to 
distinguish between exampleff from 
different classes 



*The more I terns there are providing comparable evidence, the more dependable 
the diagnosis is. 





e.g , 


e.g. , 




-''W^en 6eJLzatLng a vzn.b, hol^) maid 
you iteot i/ieie noani>: 
[1] wo.^eii, and (2) womert?" 


-Studeyit 4af/4, "I ^oaA \i}omen -c6' 


EXAMPLES 


X thz -iome mif d^i^^o^nt locm 






-"We^e o/ce Xwo noana; mafee up Xwo 
6^nt^na2^, eac/i one uting one, OfJ 
the. noun6 and the cjQfUiecX {om o^ 
the vQJib *to be^i (M mman, and 
[2] w;omen." 

The woman is pretty. 


-(jiJhen gA^ven addAjtionaZ ey.asnpleA 
lingaijOiA. and pluJtal noan&, the 
student tncoM.e(Mvf ^<uj6 "4(irne" 
Oft "cU.iSj5e/ten;C" ^hen ^tnguZoA and 
ploAol oAz compoAed ok u)hen two 
0|5 the 6ame mmbeA oJiz paJjied. 




The women is all here. 
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IDENTIFICATION 
MATRIX 



EVIDENCE SYMPTOMATIC OF CONTENT DIFFICULTIES: 
GENERALIZATIONS* 



SOURCE 
OF EVIDENCE 



OBSERVATION 
as a source of evidence about 
content difficulties 



CRITERIA 



-Commits an error on a program 
problem or task which involves 
only ^^generalization" (e,g*j a 
same/different type problem 
involving INPUTS from the same 
class) 

'On a given problem^ mere than one 
INPUT from the same class is 
presented; and the student takes 
a different ACTION in response 
to them 



RESPONSES TO PROBING 
as a source of evidence about 
content difficulties 



IN RESPONSE TO GENERAL PROBES 

'Attributes membership for the 
example of an INPUT (which was 
used in the example) to the wrong 
class 

'Attributes properties to an 
example which it does not have 
(i,e,, they are properties of 
another class) 

'Says he doesn't know which class 
the example belongs to 

IN RESPONSE TO SPECIFIC PROBES 

-When presented additional INPUT 
examples (from same or different 
classes) and asked are they the 
same or different ^ fails to 
distinguish between examples from 
different classes 



*The more items there are providing comparable evidence, the more dependable 
the diagnosis is. 



EXAMPLES 



O ■ 

ERJC 



e.g. , 

-"When ^qZojcMjiq a vc/tb, fioco mvitd 
you t/izat th^t tu)0 noan&' 
(J) man, and [2] mman?'' 

-"He/LP. oAt ^0 noun6; mafee up Xn)o 
6zntznc^6, eac/i one ii6ing one 
>t/ie nouns and tkz cohAzck, f^om 0(( 
tko, veAb 'to be': (I) looman, and 
(21 man." 

The man is s ick. 

The woman are sick. 



e.g 

because 'toomen 



women ^ 



-Whtn QA^^Jcin additional zxamptts of^ 
lAJiqaJLa/i and pZu/iaZ nOijLn6, thz 
htixdZYit incoHA^My 6ay6 "4ame" 
ox ^^dlifi^mt" w/ien 6AngaZaJi and 
pluJuxJi aJiz compaxzd ofi w/ien t^o 
0^ tkz 4ame number oJio, paiAzd, 
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J. 1.3 

IDENTIFICATION 
MATRIX 



EVIDENCE SYMPTOMATIC OF CONTENT DIFFICULTIES; 
ASSOCIATIONS* 



SOURCE 
OF EVIDENCE 



OBSERVATION 
as a source of evidence about 
content difficulties 



RESPONSES TO PROBING 
as a source of evidence about 
content difficulties 



CRITERIA 



^Commits an error* on a program 
problem or task which involves 
only "assooiation" (e.g., a 
problem in which all INPUTS and 
ACTIONS are given and the student 
is required to match them up) 

'On a given problem^ INPUTS from 
more than one class ore presented 
and the student takes a different 
ACTION for each (therefore making 
a correct discrimination) but 
matches up INPUTS and ACTIONS 
incorrectly 



IN RESPONSE TO GENERAL PROBES 

Says he knows what all the 
possible ACTIONS are (and can 
identify them) but adds that he 
doesn't know which one goes with 
uhich class of INPUTS 

'Says he took the ACTION he did 
because the example in the 
vroblem is of the type calling 
'for that ACTION 

IN RESPONSE TO SPECIFIC PROBES 

^When given all the INPUT classes 
and all the possible ACTIONS^ 
pairs them up incorrectly 



*The more items there are providing comparable evidence, the more dependable 
the diagnosis is. 



EXAMPLES 




e.g. , 

-"Matdi up tho^Q, vznb6 and noum: 



. It 



B man 
A man 



A. 



-"Mafee up a 4en^ence ^ok zach 
the^z nouyu> and a6e ihz co^AZct 
ionm oi thu vznb ^ to be': (J) tkz 
man, and (2) thz boy^.^^ 

The man a re all right. 

The boys is here today. 
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e.o . , 

-"I knoiM 4ome YioaM get '-06' a)td 
4ome get 'o/ie, ' bat I don^t knovo 
iA)hztheA. tt'6 thz 6inguJia/L noun6 
on. thci pZuAoZ noun^ tjohich get 
one o^'i thz othzA." 
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IDENTIFICATION 
MATRIX 



EVIDENCE SYMPTOMATIC OF CONTENT DIFFICULTIES; 
CHAINS 



SOURCE 
OF EVIDENCE 



OBSERVATION 
as a source of evidence about 
content difficulties 



CRITERIA 



-The student omitted a Sub-STEP 
in the chain 



'The student performed all the 
Sub-STEPS in the chain but in 
the wrong sequence 



'The student performs a Sub^STEP 
which is not a part of the chain 



RESPONSES TO PROBING 
as a source of evidence about 
content difficulties 



11^ RESPONSE TO GENERAL PROBES 



'Indicates he forgot what do do 
at a particular point 

'Indicates that he was confused 
about which Suh-STEP followed 
at a particular point 

IN RESPONSE TO SPECIFIC PROBES 

'When SiLb-STEPS are identified 
for him J he incorrectly 
identifies the required sequence 





e.g. , 


e.g. , 




-Thz mvbdlc 6tud^nt f^cuZzd to makz 


-"I don*t KmmbeA what corner 






mxt. " 


EXAMPLE 


-The moA^c student pZaifzd thz 


-"I dow'^t knoijo wkich one OiJ thz 




'yn.zpzaX}[ but cut thz m.onQ pZacz 


vohixitioYis should be plaijzd 




-in the, mu^tc 


/light novo.'' 




-The mu/itc Atadzrvt make^ a 




o _ 


"^.zpzat" wlvich l6 not callzd ion. 
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JOB PROCEDURES 





page 


SUMMARY OF PROCEDURES 


60 


Adequacy of procedures for 
interpreting learning failures 


61 




















J. 1.3 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED IN INTERPRETING 
STUDENT DIFFICULTIES OCCURRING ON PROGRAM PROBLEMS OR TASKS 



#1 



For each program problem 
on which an error is made, 

REVIEW 

a. Observational data (obtained 
from students who were 
interviewed ofter completing 
the whole instructional 
sequence) for: 

••Type of problem and 
type of error made 

••Number of students 
making errors 

b. Probing data obtained from 
al 1 tryout students 



#2 

For each program problem 
on which an error is made, 

IDENTIFY 

a. The probable type of content 
diff icul ty: 

• 'Discr iminat ions , 

••General izations, 

••Associations , OR 

• •Chains 

b. The properties of INPUTS or 
ACTIONS which are the source 
of the di ff icul ty 

c. The probable type of problem 
format di f f icul ty: 

• •Vocabulary 

• • Instructions 

••Statement of objectives 

••Other cues 
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STANDARDS 
MATRIX 



ASSESSING THE ADEQUACY OF PROCEDURES FOR INTERPRETING 
STUDENT DIFFICULTIES ON PROGRAM PROBLEMS OR TASKS 



PROPERTIES 



COMPREHENSIVE 



-Interpretation of error (s) on a 
progrcon problem attends both to: 

••Content difficulties 

CRITERIA I ••Format difficulties 

"Content difficulties rcre^ in 
addition^ further explored for: 

• -Properties of INPUTS and/or 
ACTIOnS which account for tlie 
difficulty 

-Content difficulties are explored 
to determiyie whether: 

• •No learning has occurred 
OH 

• •Incorrect learning has occurred 



DATA- BASED 



-Brass 




-Interpretation is based on data 
obtained from the entire tryout 
sample: 

• -More weight is given to a 
diagnosis based on what most 
of the tryout sample does or 
says 

-Interpretation is based on all 
available data about a program 
problem: 

Observational data 

• * Probing data 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


A program which has been revised on the basis of 
try out results. 


WHAT YOU WILL 
WORK FROM 


(1) The original program 

(2) V/eaknesses in the program which you have Identified 
on the basis of the tryout 


WHAT YOU WILL 
DO 


Revise the program 


FORMS YOU WILL 
USE 


None 



o 
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DESCRIPTION OF Sub-STEP 



INPUT 



Identified weakness 
in program 



ACTION 



Revise the program 



XI 



OUTPUT 



Revised program 

(ready for 
fiirther tryout) 



XI 1 



CRITERIA FOR 
IDENTIFYING INPUTS 



Job Aid Contents 
ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: Types 




-MATRIX: Rev is 


ing 




-MATRIX: Adequacy 


SUMMARY OF 




of program 




program 


63, 


71 


of program 


PROCEDURES . . 


. . 86 


weakness . . 67 , 


69, 


-MATRIX: Rev is 


ing 




revision .... 87 








70 


for specific 
















diff icul ties . 
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-MATRIX: 
















Priorities to 
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revi s ing 
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Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Interpretation of 
learning failures 


J. 1.3 


FORM J. 2(1) carried 
forward from 


J. 1.3 
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Sub-STEP 



J. 1.4 



INPUT 



Identified weakness 
in program 



JOB DIAGRAM 



ACTION 



Revise the program 



XI 



OUTPUT 



Revised program 

(ready for 
further tryout) 



xii 



Difficulties 
in isolated, 
individual problems 



x.a 



Revi se 
individual p rob 1 ems 



XI .a 



Change in: 
-Fo rmat 
-Content 



XI I .a 



Difficulties 
in a cumulative 
series of problems 




Revi se 
series of problems 


^ 


Change in: 
-Format 
-Content 




► 






•'Learning of program 

••Assistance for 
speci f i c ski 1 1 s 


x.b 




xi ,b 




xi 1 , b 
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BACKGROUND INFORMATION 





page 


Extent of program weakness: Errors 
on individual vs. cumulative series 
of program problems 


66 


Sources of program weakness likely 
to be associated with programs which 
are weak to differing extents 


67 


How to revise individual prog ''am 
problems which result in errors 


68 


Two sources cf program weakness 
associated with errors on a series 
of program problems 


69 


Four types of program "leanness" 
which may account for errors on a 
series of program problems 


70 


How to revise program problems 
to make them less lean 


71 


Criteria for determining priorities 
in the kinds of revisions which are 
made 1 n the i ns truct lonal program 


72 


Priorities in revising a series of 
program problems in '^rder to n;jke 
them less 1 ean 


73 
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J. 1.4 

IDENTIFICATION 
MATRIX 



EXTENT OF PROGRAM WEAKNESS 



EXTENT OF 

PROGRAM 
WEAKNESS 


Errors are made on 
INDIVIDUAL 
program problems 


Errors are made on a 
CUMULATIVE SERIES 
of program problems 


CRITERIA 


-An informal record of errors or 
F0m4 J. 2(1) filled out reveals 
that errors are made on: 

••Isolated program problems or 
tasks 

••Problems involving different 
skills within the same portion 
of a cliain 

• •Problems involving widely 

• •Problems involving preparatory 
behaviors or criterion 
behaviors but usually not both 


-An informal record of errors or 
FORM J. 2(2) filled out reveals 
that errors are made on: 

• •A sequential scries of program 
problems 

••Problems involving different 
examples related to the saw<^ 
skill being taught 

••Problems involving adjacent 

••Problems usually involving both 
the criterion behaviors and 
preparatory behaviors 

i 
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J. 1.4 

IDENTIFICATION 
MATRIX 



SOURCES OF PROGRAM WEAKNESS LIKELY TO BE ASSOCIATED 
WITH ERRORS ON INDIVIDUAL PROGRAM PROBLEMS 
AND ON A CUMULATIVE SERIES OF PROGRAM PROBLEMS 



EXTENT OF 
PROGRAM 
WEAKNESS 



CRITERIA 



Sources of progr^'.. weakness 
likely to be assoclcted with 
isolated INDIVIDUAL 
program probleo^s 



-Program FORMAT 

••Vocabulary too diffiault 

••Instructions unclear or 
confusing 

• • Attention- controlling 
techniques inadequate 

-Program CONTENT 

• •Assistance provided (i,e, ^ 
cues) makes an inadequate 
identification of INPUTS or 
ACTIONS required for successful 
vrohlsm solution 



Sources of program weakness 
likely to be associated with a 
CUMULATIVE SERIES 
of program problems 



'Program CONTENT 

••The instructional sequence is 
e^ccessivelij lean 

• •FrereqiA.isite behavior is wrongly 
assumed to he in student *s 
repertoire 

••Prerequisite behavior to he 
taught in the program is taught 
in the wrong sequence (i.e.y 
not taught before the criterion 
behavior in question as it 
should be) 

• •Prerequisite behavior to be 
taught in the program is not 
adequately taught 



ERLC 



67 



J. 1.4 

DECISION 
MATRIX 



DETERMINING HOW TO REVISE AN INDIVIDUAL PROGRAM. PROBLEM 
ON WHICH ERRORS ARE MADE* 



CONDITIONS 



Error Is due to 
FORMAT characteristics 
(as revealed In Sub-STEP J. 1.3) 



Error Is due to 
CONTENT characteristics 
(as revealed In Sub-STEP J. 1.3) 



ACTION 
TO TAKE 



-Change Voccdnilary level and 
readability level to make it more 
suitable to the target audience: 

••Shorter, more commonly used 
words 

• •Shorter sentences 

-Change problem or task 
instructions making them: 

• •Clearer, less ambiguous 

••Easier to understand 

-Change attention or observation 
controlling devices to make them 
work more effectively: 

••Simplify the devices 

• •Make features more prominent 



'Increase the strength of \ 
assistance: 

• •Make the identification of 
INPUTS ,md ACTIONS more direct 

/If no identification is made, 
provide an indirect 
identification 

/If identification is already 
indirect, make it less 
i direct 

/Only in a beginning problem 
in a series should an indirect 
identification be replaced by 
a direct one (See SECTION G, 
pages 90-95) 



*When two or more of the Informal tryout sample (of five or six students) make an 
error on the problem 



EXAMPLES 



ERIC 



e.g. , 

Informal tryout has revealed that 
students did not know the meaning 
of a (non-technical) word and a 
word not included In the 
objectives for the Instructional 
sequence 

Onxglnal mnd a6Q,d u;ai 
*'ema^axlate,." Ckango, to 'Wafeen" 
OK "txnnc/Lve. " 

e.g. , 

A diagram was used to control the 
observing behavior of the student; 
students did not know how to use 
it 

C/iange in6t/uLction& 6o that 
\tudQ,nt6 knoi^) lokeAQ, to ayit^ tka 
dyiaqKom, lohat to look at ivUit, 
ioliax to look j^oA., t^kat to do noxt, 

2XC. 
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e.g. , 

The practice problem offers no 
identification of the ACTION 
(which is required) 

Tfie ^tadant ajs Ko^qavidd to oAe a 
veAb [otheA than tho6Q, oAed -in 
pnloK Q,xampl<i^) and to iK^a tiia 
conAQ,ct ionm {^ok tho, i^abjact noun 
qivoyi hJjn, No a^^^tancQ, t& givm 
kirn, 

'On KtVAJiion fie givm the. 
indOiQ,ct Ajde,ntLfiicatxon--anoth£A 
noun ayvd anotkoA veAb'- "Tfie man 
{A)aZk6 to lOOKk," 

'In Itu nm pn,oblm tMk ha ^ 
givQ.n tiia Mibjo^ct noun "t/ze dog" 
and AJi fiQ^qiJUAtd to make, up a 
sentence -tn the, ji^e^mt to^n^o. oA^ng 
anothoA voJib, Tfie axainpla givon 

A^dVi^dt OK poAtixil ido^ntlilaaXyion 



J. 1.4 

IDENTIFICATION 
MATRtX 



TWO MAJOR TYPES OF PROGRAM WEAKNESS 
ASSOCIATED WITH CONTENT DIFFICULTIES 
IN A SERIES OF PROGRAM PROBLEMS 



TYPES OF 

PROGRAM 

WEAKNESS 



EXCESSIVE 
LEANNESS 



'The instructional program which 
has been prepared is too lean 
resulting in content difficulties 



CRITERIA 



PREREQUISITE BEHAVIORS 
not in student's repertoire 



^Student difficulties are due to 
the absence from the student's 
repertoire of other criterion 
behaviors which are prerequisite 
to the learning of the criterion 
behavior in question 

^This may be due to one of tDo 
conditions : 

- •Students did not hm>e 
prerequisite behaviors to begin 
with as may have been assumed 

•'If the prerequisite behavior 
is taught in the program j its 
sequencing relative to the 
criterion behavior in question 
is at fault 



FOR DETAIL 



o ■ 
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See next page 
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-Analysis of this type of 
weakness requires the use of 
data not available during i n formal 
try out 

-Sequencing changes are therefore 
made during developmental tryout 

See STEP J. 2 
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J. 1.4 

IDENTIFICATION 
MATRIX 



CRITERIA FOR DETERMINING PRIORITIES 
IN WAYS TO REVISE INSTRUCTIONAL SEQUENCES 
IN ORDER TO MAKE THEM LESS LEAN 



VARIABLES 



EFFICIENCY 
of an instructional sequence 



CRITERIA 



-Revision should be consistent 
with the efficiency of the 
instructional sequence; the 
sequence is more efficient: 

••The shorter it is (i,e.j the 
fewer problems there are) y 

••The less time it takes to 
complete 

-Revision should keep the 
instructional sequence as lean 
as is possible J consistent with 
''EFFECTIVENESS" 

-Revision should not add any more 
than is required 

-Accordingly y each cycle of 
revision snould make minimal 

changes away from leanness 



EFFECTIVENESS 
of an instructional sequence 



"Revision must result in an 
instructional seqixence which now 
produces minimal number of errors: 

• •Few program problems produce 

errors 

• •Few students out of total 

try out population make errors 
on program problems 

-Revision should be geared to the 
type of learning difficulties 
occtivring — but consistent with 
the criterion of "EFFICIENCY" 



72 



ERIC 



J. 1.4 
DECISION 
MATRIX 



DETERMINING PRIORITIES AMONG THE FOUR WAYS 
TO REVISE PROGRAM SEQUENCES TO MAKE THEM LESS LEAN 



PRIORITIES 



ACTION 
TO TAKE 



1st priority 
occurring in thf. 
FIRST 
revision cycle 



'Keep "the AMOUNT of 
criterion behavior 
practiced at the 
same time" unchanged 

'Considei* one or more 
of the fotlouing 
approaches: 

• * Strengthen the 
degree of assistance 
to the program 
problems already in 
the sequence and/or 
remove assistance 
less early in the' 
sequence 
OR 

' * Introduce new 
program problems to 
the sequence provid- 
ing more examples 
(new or repetitions) 



2nd priority 
occurring In the 
SECOND 
revision cycle 



IF THE SECOND 
TRYOUT STILL PRODUCES 
ERRORS ON A SERIES 
OF PROGRAM PROBLEMS: 

-Keep "the AMOUNT of 
criterion behavior 
practiced at the 
same time" unchanged 

-Definitely add new 
program problems 
appropriate to the 
skills with which 
students have 
difficulties. 
(Intersperse the new 
problems among the 
existing problems) 

-Also consider 
strengthening the 
degree of assistance 
for existing problems 



3rd priority 
occurring In the 
THIRD 
revision cycle 



IF THE THIRD 
TRYOUT STILL PRODUCES 
ERRORS ON A SERIES 
OF PROGRAM PROBLEMS: 

-Reduce the amount of 
criterion behavior 
practiced at the same 
time (one level lower 
on each successive 
revision) 

-Also consider adding 
new program problems 



EXAMPLES 



SEE SECTION "I/' PAGES 7^-96 
FOR EXAMPLES OF PROGRAM SEQUENCES 
DIFFERING IN "AMOUNT" OF CRITERION BEHAVIOR 
PRACTICED AT THE SAME TIME 



ERIC 



REVISING PROGRAM PROBLEMS 
FOR SPECIFIC TYPES OF LEARNING DIFFICULTIES 





page 


How revising "assistance" and 
"number of program problems" is used 
in order to overcome specific learn- 
ing difficulties with individual 
program problems either "isolated" 
or "part or a series of problems" 


76, 77 


Ways to revise program to overcome 
"discrimination" difficulties 


78 


Ways to revise program to overcome 
"generalization" difficulties 


79 


Ways to revise program to overcome 
"association" difficulties 


80 


Ways to revise program to overcome 
"chaining" difficulties 


81 




HOW REVISION OF "DEGREE OF ASSISTANCE" AND 
OF "NUMBER OF EXAMPLES". IS USED TO OVERCOME 
DIFFICULTIES IN A PROGRAM PROBLEM* 



TYPE OF 
REVISION 


STRENGTHENING 
THE DEGREE OF ASSISTANCE 


ADDING 
NEW PROGRAM PROBLEMS 


CRITERIA 


'Cues or prcmpts are added which 
provide move diveot identifiaa- 
tiorw of' 

'•INPUTS/OUTPUTS to be 
discriminated and/or the 
properties which form the basis 
for the discrimination 

INPUTS/OUTPUTS recfuiring 
generalization and/or the 
properties which form the basis 
for the generalization 

* -Which ACTION goes with which 
INPUT (associations) 

••f}.e sequence of associations 
to be chained 


'New program problems are added 
or old problems are repeated 
giving the student additional 
practice : 

••Discriminating between new/old 
examples of INPUTS (or OUTPUTS) 

• •Generalizing across new/old 
examples of INPUTS (or OUTPUTS) 

••Associating new/old examples of 
INPUTS with ACTIONS 

• •Producing a chained seQuen^e 
of associations 



*These revision approaches apply both to Isolated problem problems (on which errors 
occur) or on program problems which are part of a series 



EXAMPLES 



SEE PAGES 77-81 



ERIC 
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J-1,4 

IDENTIFICATION 
MATRIX 



TWO APPROACHES TO CONSIDER IN MAKING REVISIONS 



APPROACHES 



Use the SAME 
practice progression 
selected in TASK "G" 
appropriate to 
the learning difficulties 



CRITERIA 



-The strengthening of assistance 
or the addition of new examples 
is done within the framework of 
practice progressions selected 
in TASK "G" 

SEE PAGES 71-1^9 

IN SECTION "G" 
OF THE HANDBOOK 



Consider using NEW 
practice progression 
appropriate to 
the learning difficulties 



'The strengthening of assistance 
or the addition of new examples 
is done within the framework of 
new practice progressions 
selected now as a means of 
reducing learning difficulties 

'The new progressions are selected 
because of empirical evidence 
that the difficulty is greater 
than initially anticipated 



e.g. , 

-T/ie pHjoQKQ/i^lon 6tlz(itejd 
GAJj^^YiaZZy 'XcUi a "mode" 
pKoQK^&^on -invotv-Lng only zd/Jxm 
EXAMPLES and pKodaati -on p^cticz; to add to 

thxA, /LZcognAXion p^acticz mi.ght 
aZ(>o fae lnclude.d to mafee thz 

-A ^cqaence davotzd to pn^actLcz of^ 
dAJiCAAjmX.natiorU> A^YvUxaJctij tt6ed 
poAAfid exampie/6 — fj^AAt d^/J^cAiyig 
wldoZvi^ tlim gAadaaZiy na/oiownd, 
Thz KzvA^ton might ZncZadz 
addLtionat pKobZejn6 in u)kich the. 
/Lzdaction in di^^QAmce^ bztwzzn 
pcuM if> ^tilt moKZ QKadiiaJi 



e.g. , 

-T/ie onigtnat 6zquznct macf havz 
p^ovtdzd ontij f^o/i production 
p^acticz [no modz pKogKOJ^^ion 
in\)oZ\)zd] • The addition o{i 
n^zdogyiityion and zditing pn.a<iti(it 
i6 now htlzctzd bzcaa^z o^ thz 
dmon^t^cLtzd di^icuZti^f o(i thz 
ZzoAning p^obZm 



77 



J. 1.4 
IDENTIFICATION 
MATRIX 



WAYS TO REVISE A PROGRAM SEQUENCE 
AS A MEANS OF OVERCOMING "D I SCRI M I NAT I ON'' DIFFICULTIES 



REVISION 
APPROACHES 



Adding ASSISTANCE 
wi thin an al ready 
selected progression 



Adding PROBLEMS 
wi thi n an al ready 
selected progression 



Using other 
progressions 
which are appropriate 



CRITERIA 



-Providing as a model 
contrasting examples 
of INPUTS which belong ^ 
to a class and those 
which do not 

'Providing as a model 
contrasting exoifiples 
of INPUTS which belong 
to different classes 

'Providing verbal rules 
which, identify when 
one class begins and 
another leaves off 

-Using cues (visual or 
verbal) calling 
attention to proper- 
ties which determine 
whether INPUTS belong 
to one class or 
another 



'Providing additional 
program problems in 
which new examples of 
INPUTS from different 
classes are used, (and 
about which the 
student practices 
making relevant 
discriminations ) 

'Providing additional 
program problems 
which repeat the use 
of examples already 
used 



"Providing recognition 
practice in which 
pairs of examples 
belonging to 
different classes and 
to the same class are 
contrasted and /or 
compared 
(Progression b) 

-Using diagrams to 
highlight the 
differenc es^ between 
classes 

(Progression i) 

'Using widely 
contrasting examples 
and. gradually 
narroti)ing differences 
to permit fine 
discrimina ti ons 
(Progression k) 



^These two approaches arc often comb i ned . 



FOR 
EXAMPLES 
SEE PAGES 



In SECTION "G" 



in SECTION "G" 



in SECTION "G" 
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J J,4 
IDENTIFICATION 
MATRIX 



AS 



WAYS TO REVISE A PROGRAM SEQUENCE 
A MEANS OF OVERCOMING "GENERALIZATION" DIFFICULTIES 



REVISION 
APPROACHES 



CRITERIA 



Adding ASSISTANCE 
within an already 
selected progression 



-Providing as a model 
contrasting examples 
of INPUTS Which belong 
to a class and those 
which do not 

'Providing as a model 
pairs of comparable 
INPUTS which belong 
to the sa)ne class 

'Providing verbal rules 
which identify when 
one class begins and 
another leaves off 

-Using cues (visual or 
verbal) calling 
attention to proper- 
ties which determine 
whether INPUTS belong 
to one class or 
another 



Adding PROBLEMS 
within an already 
selected progression 



'Providing additional 
program problems (in 
which new examples of 
INPUTS from the same 
class or 3 used (and 
about which the 
student practices 
making relevant 
generalizations) 

'Providing additional 
program problems 
which repeat the use 
of examples already 
used 



Using other 
progressions 
which are appropriate 



-Providing recognition 
practice in which 
pairs of examples 
belonging to 
different classes and 
to the same class cere 
contrasted and/or 
compared 
(Progression b) 

-Using diagrams to 
highlight the 
s imi larities within a 
c'iass 

(Progression i) 



*These two approaches are often combined 



FOR 
EXAMPLES 
SEE PAGES 



ERLC 



9^-95 
SECTION 



109-113 
SECTION 



"G' 
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See Sub-STEP 
in SECTION 



G.2.1 



J. 1.4 

IDENTiFICATION 
MATRIX 



AS 



WAYS TO REVISE A PROGRAM SEQUENCE 
A MEANS OF OVERCOMING ^'ASSOCIATION'* DIFFICULTIES 



REVISION 
APPROACHES 



Adding ASSISTANCE 
wi th in an al ready 
selected progress ic 



CRITERIA 



-Providing as a model 
an example from each 
INPUT class and the 
ACTION to be 
associated jrith it 
(to he doixe for all 
classes) 

-Providing a verbal 
rule indicating how 
INPUT and ACTIONS 
are associated 



Adding PROBLEMS 
within an already 
r.?-l»?cted progression 



'Providing additional 
program problems with 
which the student 
gains further 
practice in associat- 
ing INPUTS and 
ACTIONS: 

• •Repetition of old 

(INPUT) examples 
OR 

• • New ( INPUT) examp le £ 



Using other 
progressions 
which are appropriate 



-Providing recognition 
practice in which:, 
when given an INPUT ^ 
the student merely 
has to select a 
correct ACTION from 
two or more options 
(Progression h) 

'Using diagrams to 
give an overview of 
which ACTION goes 
with which INPUT 
class (Progression i) 



*These two approaches are often comb 1 ned . 



FOR 
EXAMPLES 
SEE PAGES 



In SECTION "G^ 



109-113 
in SECTION "G" 



See Sub-STEP G.2.1 
in SECTION "G" 
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IDENTIFICATION 
MATRIX 



WAYS TO REVISE A PROGRAM SEQUENCE 
AS A MEANS OF OVERCOMING "CHAINING" DIFFICULTIES 



REVISION 
APPROACHES 



Adding ASSISTANCE 
within en al ready 
selected progression 



-Providing a model 
perfomanoe of the 
whole chain which the 
student can imitate 

-Providing verbal 
rules indicating' the 
proper sequence in 
which the separate 
associationo are 
chained 

-Providing verbal or 
visual cues which 
help identify which 
association is next 
up in the sequence 



Adding PROBLEMS 
wi th i n an al ready 
selected progression 



-Providing additional 
practice in producing 
the whole chain 



Using o ther 
progress ions 
which are appropriate 



'Providing recognitio n 
practice in which the 
student has to select 
from options which 
association comes 
next in the chain 
(Progression h) 

'Using diagrams to 
give an overview of 
a conceptual chain 
or a flow pattern of 
a procedural chain 
(Progression i) 



^These two approaches are often comb ined . 
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JOB PROCEDURES 





page 


Information to review 
and what to look for 




Deciding how to approach 
revision 


85 


SUMMARY OF PROCEDURES 


86 


Adequacy of appropch 
to revision 


87 
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J. 1.4 
IDENTIFICATION 
MATRIX 



INFORMATION SOURCES TO REVIEW AND WHAT TO 
LOOK FOR AS A BASIS FOR PROGRAM REVISION 



SOURCES 


OBSERVATIONAL DATA 
recorded on actual program 
AND 

recorded on FORM J.2(l) 
and interpretation of 
student difficulties 
in Sub-STEP J.K3 


PROBING DATA 
obtained in Sub-STEP J. I. 2 
AND 

interpretation of 
student difficulties 
in Sub-STEP J.l .3 


i 


1 -'Ident'if'iccit'ioTi of extent of 
student errors: 




CRITERIA 


••On individual^ isolated 
program problems 
OR 

• •On a cumulative series of 
program problems 








"TdpYit'L'Fit^titinn of* • 




••Type of student difficulties 


••Type of student difficulties 




••Sources of the difficulties: 


• •Sources of the difficulties: 




/Content difficulties 


/Content difficulties 




/Format difficulties 


/Format difficulties 



8ii 



DETERMINING HOW TO DEAL WITH WEAKNESSES IN PROGRAM: 
ON INDIVIDUAL PROGRAJi PROBLEMS 
AND ON A SERIES OF PROGRAM PROBLEMS 



CONDITIONS 


Student errors occur 
on individual. Isolated 


Student errors occur 
on a cumulative series of 


program problems 


program problems 


ACTION 
TO TAKE 


-Make format changes appropriate 
to identified difficulties: 

"•In vocabulary J or in sentence 
complexity^ etc, 

**In ambiguity of task 
instructions 

••In ambiguity of attention or 
observation controlling devices 

-Make content ohanqes appropriate 
to type of content difficulty 
(i,e,j type of component skill 
not learned): 

••Improve old progressions used 
or use neb) progressions 

• 'Strengthen assistance or 
provide new examples 


'■ . 1 

-Make content changes appropriate 1 
to extent of problem; decrease 1 
leanness of program by: 

• •St^-'erigthening assistance 

• • Adding program problems 

••And J on final cycles of 
revision J reduce the amount of 
criterion behavior practiced 
at the same time 

"Within the series y for each 
program problem make content 
changes appropriate to type of 
content difficulty (i,e.j type of 
component skill not learned): 

• •Improve old progressions used 
or use new progressions 

• •Strengthen assistance or 
provide new examples 
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J. 1.4 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED IN IDENTIFYING 
PROGRAM WEAKNESSES AND MAKING APPROPRIATE REVISIONS 



b. 



REVIEW 

Observat ^onal data 
recorded on 
FORM J. 2(1) 

Observational data 
and probing data 



n 



IDENTIFY 

Extent of program 
weakness 

Type of student 
difficulty and the 
sources of the 
dl ff icul ty 



#3 

REVISE 
instruct iona) sequence 

a. Reflecting the 
di ff Icul ty 
identified for 
each program 
problem 

and/or 

b. Reflecting the 
dif f Icul ty in a 
cumulative series 
of program 
problems 



USSON I 
Opli(m fiHlarwd 



FORM J.2(l) 



SUMrtARY Of iBROR.'i ON CFITifllON TiST 




..L. L- 



i I 
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J. 1.4 

STANDARDS 
MATRIX 



PROPERTIES 



ASSESSING THE ADEQUACY IN APPROACH 
TO REVISION OF IDENTIFIED PROGRAM WEAKNESSES 



1 

DATA- BASED 



EFFECTIVENESS 



EFFICIENCY 



CRITERIA 



-Revisions made are 
closely geared to 
errors identified 
from 

-Observational data: 

** Errors on isolated 
program problems 

• * Errors on a 
cumulative series 
of program problems 

-Probing data and 
observational data: 

Errors in learning 
specific component 
skills (content 
difficulties) 

Format difficulties 

/Vocabulary 

/Instructions 

/Attention- 
controlling devices 



^Techniques used for 
revision consist of: 

Using progressions 
originally selected 
for specific 
learning problems^ 
OR selecting new 
ones when errors 
persist on 
cyclically try outs 

• •Reducing leanness 
of program by 
adding : 

/Stronger assistance 

/More practice 
prob lems 

-The instructional 
sequence in its final 
cyclical revision is 
relatively error- free; 
i.e. J errors are made 
on fewer than 25-20% 
of the program 
problems 



^Revisions ave made 
in such a way as to 
keep the instructional 
sequence as lean as 
possible J compatible 
with the goal of 
producing an 
effective sequence: 

••The amount of 
criterion behavior 
practiced at the 
same time^ only as 
a last resort after 
other revision 
strategies (See 
Column #2) fail in 
early revisions 
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STEP 



COMPLETION CHECKLIST 



IDENTIFIED 


PERFORMED 


PRODUCED 


FORMS COMPLETED 


Enter proficiency 


Administration of a 






of informal tryout 


"before" test 






samp I e 









J. 1.1 





Errors on program 


Administration of 




FORM J.2(l) 


J. 1.2 




instructional 




(optional) 






program 









Interpreted nature 
of learning fai lures 












Revi sed vers ion of 
program 





J. 1.3 



J.l .4 
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ERIC 



STEP 



J. 2 



J. 2 



Conduct a developmental tryout of the instructional program and re,;se 
the program when necessary.* 



J. 2.1 



Administer the criterion and sub-criterion tests to a representative 
sample of the target population as a "before" test. 



J. 2. 2 



Administer the instructional proF^ram to the representative sample 
and identify the portions of the program on which errors are made. 



J. 2. 3 



Administer criterion and sub-criterion tests to the representative 
sample as an "after" test, identify errors, and, whenever possible, 
interpret learner difficulties which have resulted in errors. 



J. 2. 4 



Administer diagnostic tests and/or conduct interviews to identify 
learner difficulties which have resulted in the errors which have 
been made . 



J. 2. 5 



Make revisions in the instinictional program based on ^program and 
test error data and on identified sources of errors. 



ERLC 



*The cycle of Sub-STEFS J. 2.1, J. 2. 2, J. 2. 3, J, 2. 4, and J. 2. 5 should be 
repeated until a predetermined standard of performance is attained by 
a predetermined percentage of a representative sample. 



OVERVIEW 



INPUT 



Instructional program 
cyclically revised 
during Informal tryouts 
+ 

tests 
+ 

sample audiencf^ 



ACTION 



(Cyclically) 
Administer program, 
identify weaknesses, 
and revise program 



OUTPUT 



Final version of the 
instructional program 

which meets 
predetermined standards 



J. 2.1 



Target audience 
+ 

criterion and 
sub-criterion tests 



J. 2. 2 



Sample of students 
selected for 
developmental try out 
+ 

instruct! on aJ. program 
(revised during 
informal tryout) 



J. 2. 3 



Tryout sample 
which has completed 
instructional program 
and 
tests 

vii 



J. 2. 4 



Analyses of 
program eri'ors 
and 
test errors 



J. 2. 5 



Identified sources 
of errors 
on program 
and on tests 



xii: 



ERIC 



Select representative 
s ample , 
administer tests 
to them as a 
"before" test, 
and score tests 



Administer program 
+ 

observe errors as they 
occur (for proceduraJ. 
program , OR 
score errors at end 
of administration (for 
paper-and -pencil v 
programs I 



Administer tests 
as an "after" test 
and 
score tests 
and 

interpret errors 

viii 



Administer 
diarrnostic tests and 
conduct interviews 
when necessary 

xi 



Identify 
program weaknesses 
and make 
appropriate revisions 



XIV 
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Sample of 
target t^udience 
+ 

identification of 
how much of 
criterion behavior it 
already possesses 
on FORM J. 2(1) in 



Identification of: 

-Portions of program on 
which errors are made 
(on FOm J. 2(1)) 



VI 



Identification of 
how much of criterion 
and sub-criterion 
behaviors the sample 
now possesses 
and 

interpretation of source 
of errors committed ix 



Identification of 
types of learning 
failures 



Revision in: 
-Isolated problems 
-Series of problems 
-Total program 
-Sequences ^cv 



STEP 



J, 2 







PAGE 


INDEX 






CRITERIA FOR 










IDENTIFYING INPUTS 


ACTION TO BE TAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 






-MATRIX: How to 


-MATRIX: Adequacy 


FORM J, 2(1) 




• 


identify "before" 
proficiency • • 3(> 


of sample reaction 
and use of 
"before" test 
results . , • • 99 


SUMMARY OF 
PROCEDURES . . 98 


















-MATRIX: Adequacy 


FORM J, 2(1} 


J. 2, 2 






of program 
administration and 
i dent i f i Cot i on of 
errors • , . • 105 


SUMMARY OF 
PROCEDURES • • 10^ 



J. 2. 3 



-l\AJR 1 X : 


Assessing proficiency . . 


. Ill 


-MATRIX: 


GROUP resul ts . 


. 117 


-MATRIX: 


1 WD! VI DUAL results . . . 


. 125 


-MATR 1 X : 


Deiayed assessment . . . 


. 151 


-HATR i X : 


Assessing "sequencing" 


. 155 



•MATR-IX: Adequacy 
of interpretation 
of errors on 
"after" 

tests .... 163 



FORM J. 2(1) 
FORM J. 2 (2) 
FORM J. 2 (3) 

SUMMARY OF 
PROCEDURES 



162 



-MATRIX: 
Inforoat ion 
sources to 


-MATRIX: When 
to do further 
d i agnost i c 


"MATRIX: 
Adequacy 
of procedures 
involved in 
doing further 
di agnos t i c 


SUMMARY OF 

PROCEDURES . . 1 70 




-MATRIX: Four 
dftqrcfls of pro<jrrtri 

Idcnt 1 fy In/j 
VrtJokr>ci*c^ In 


-MATRIX: DnctfJInn 
on thn doqrno of 

rovlslon . . . . I^n 
• liftTRIX: llov/ to 
revise 

proo^ons ... iB^i, 
\%-iO\ , 207 
-HATRIX: RpvUJna 
^cqucncc^ .210-212 


-MATRIX: Adequacy 
of program 
revision . . . 217 


SUMMARY OF 
PROCEDURES . . 2l6 



J. 2. 4 



J. 2. 5 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


f/hat the target audience con already do before 
TeceitJ-^rig instruction is identified^ 


WHAT YOU WILL 
WORK FROM 


(1) The target audience 

(2} Criterion and' sub-crrterfon tests 


WHAT YOU WILL 
DO 


(1) Select a representative sample from the target 
audience (n - approximately 25 students). 

(2) Administer and score the tests (a BEFORE test). 

(3) Identify v/hat portions of the criterion behavior, 
the sample already can exhibit. 


FORMS YOU WILL 
USE 


FORM J,2(l) for recording errors on tests. 



ERIC 



DESCHIPTION OF Sjb-STEP 



J. 2.1 



INPUT 



Target audience 
+ 

criterion and 
sub-criterion tests 



ACTION 



"elect representative 
sample, 
administer tests 
to them as a 
"before" test, 
and score tests 

ii 



OUTPUT 



Sample of 
target audience 
+ 

identification of 
hov much of 
criterion behavior it 
already T>ossesses 
on FORM J. 2(1) iii 



Job Aid Contents 

CRITERIA f6r 

IDENTIFYING INPUTS ACTION TO BE T» aEN STANDARD FOR OUTPUTS FORMS TO USE 





-MATRIX: How to 
identify "before" 
proficiency . . 96 


-MATRIX: 
Adequacy of 
sample reaction 
and use of 
"before" test 
results .... 99 


FORM J. 2(1) 

SUMMARY OF 
PROCEDURES . . 98 



Required Maierials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Identification of 
audience characteris- 


J.l .1 


Tests on FORM F.2(l) 
(carried forward 
from) 


J.l .1 


FORM J. 2(1) 
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Sub- STEP ! J. 2.1 



JOB DIAGRAM 



INPUT 



ACTION 



OUTPUT 



Target audience 
+ 

criterion and 
siib-criterion tests 



Select representative 
sample , 
axiininister tests 
to them as a 
"before" test, 
and score tests 

ii 



Sample of 
target audience 
+ 

identification of 
hov much cf 
criterion behavior it 
already possesses 
on FORM J. 2(1) 



111 



A hi gh percentage of 
sampl e 
makes errors on: 

-Individual test items 

-All items related to a 
sub-criterion behavior 

-All test items related 
to the CRITERION 
behavior - 



Interpret the 
degree of 
'•entering*' or "before" 
proficiency 



Sample has 
low "before'* 
proficiency at: 

■Sub-cr i terion 
behavior(s) 

■The CRITERION behavior 



I .a 



A low percentage of 
sample 
makes errors on: 

-Individual test items 

-All Items related to a 
sub-criterion behc^vior 

-All test items related 
to the CRITERION 
behav i or 

i .b 



I nterpret the 
degree of 
"entering" or "before" 
proficiency 



i .b 



Sample has 
high "before" 
proficiency at: 

-Sub-cr I terion 
behavior (s) 

-The CRITERION behavior 
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JOB PROCEDURES 





page 


Identifying "before" proficiency 


96, 97 


SUMMARY OF PROCEDURES 


98 


Adequacy of sample selection and 
use of "before" test results 


99 
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J-2J 
DECISION 
MATRIX 



DETERMINING "BEFORE" PROFICIENCY 
BASED ON THREE TYPES OF INFORMATION 



TYPES OF 
MEASURES 



ACTION 
TO TAKE 



Proficiency on 
INDIVIDUAL 
test items 



'Record on FORM J, 2(1) 
the number of atudente 
making an error on 
each test item 

'(For multiple choice 
t?>at items y enter 
number of students 
selecting each wrong 
option) 

"Express the number (of 
students making 
errors) as a percent 
of the total sample; 
enter on "percent" 
roD of FORM J, 2(1) 



SUB 



Proficiency on 
a1 \ test 1 terns 
related to each 
-CRITERION behavior 



-Taking the values from 
the "percent" raio on 
FORM J. 2(1)^ add the 
percentages for all 
test items related to 
each sub-criterion 
behavior 

'-Divide this sum by 
the number of test 
items related to each 
sub-criterion behavior 

'This produces an 
average error score 
for each aub-criterion 
behoDior 



Proficiency on 
al 1 test items 
related to each 
CRITERION behavior 



'Tak''^,ng the values 
from the "percent" 
row on FORM J. 2(1), 
add the percentages 
for all test items 
related to the 
CRITERION behavior 

'-Divide this sum by 
the number of test 
items related to the 
CRITERION oehavior 

'-This produces an 
average error score 
for the CRITERION 
behavior 



EXAMPLE 
(See oppos i te 
page) 



•The number of people 
making an error on 
each test item is 
f i 1 led in in the f i rst 
row; e.g., 24 students 
made an error on 
Item #A • 



-Test items 10-15 all 
relate to 
sub-criterion behavior 

#2 

-The sum of the 
percentages for each 
of these items (48, 
56, 60, 36, 52, and 
kk) equals 296 

-There rre 6 I tems 

-296 divided by 6 
yields an average 
error rate of k3% on 
a) ) i tems related to 
thi s sub-cri teri on 
behavior 

-This me'ans that S\% of 
the sample is 
proficient at this 
sub-'cri terion behavior 



-Test i tems 22-25 al 1 
relate to the 
CRITERION behavic- 

-The sum of the 
percentages for each 
of these items (90, 
96 , 96, and 84) equals 
366 

-There are 4 items 

-366 divided by 4 
yields an average 
error rate of 3\% on 
al 1 I tems related to 
the CRITERION 
behavior 

-This means that 03% of 
the sample is 
proficient at the 
CRITERION behavior 
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J. 2.1 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED IN SELECTING 
A SAMPLE, ADMINISTERING AND SCORING '^BEFORE" TESTS 



SELECT SAMPLE 

a. Select sample of 
approximately 
25-50 students from 
the target audience 

b. Make sure sample 

i s representat i ve* 
(random selection 
can assure this) 

See Sub-STEP J. 2.1 
for discussion of 
sample selection. 



tt7 

ADMINISTER AND 
SCORE TESTS 

Admlnl ster 
sub-crt ter Ion tests 
and criterion tests 
to sample 

Score tests 

Record error 
results by test 
I terns on top half 
of FORM J.2(l) 

See Sub'STEP J. 1.2 
for discussion of 
use of FORM J. 2(1). 



#3 

IDENTIFY 
"BEFORE" PROFICIENCY 

a. Review scores for 
all test Items 
related to each 
iub-cr I terion 
behavior 

b. Sum results for 
each sub-criterion 
behavior 

c. Determine which 
students are already 
proficient at the 
sub-criterion 
behavior and hov; 
many of the total 
group are proficient 

d. Repeat a-c for each 
sub-criterion 
behavior and for the 
criterion behavior 



FORM J. 2(1) 



SUMMARY Of tRHORS ON CRfTeRlON JtST 



ITEM if-S Q 




2 


1 


4 


S 


1 


T 


t 


t 


10 


11 


)3 


13 


14 




IB 


1; 


tl 


It 


20 


21 


22 


23 


24 








































































































































































































































































Toi* Muf^ 










































































































1 


2 


t 


4 


s 


< 


7 


• 


ft 


10 


11 


13 




M 


M 


14 


17 


IS 


If 


20 


21 


22 


23 


24 


» 


<N*UT •«<i>l 








































































































•tTION 




















































ACTION \t*m**> 





















































^Proficiency at criterion behavior may range from zero to 100%. 
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0.2.1 

STANDARDS 
MATRIX 



ASSESSING THE ADEQUACY OF SAMPLE SELECTION 
AND ADMINISTRATION OF "BEFORE" TESTS 



PROPERTIES 


SAMPLING 


SCORING 


adequacy 


adequacy 


CRITERIA 


-Sample size: 

••Sample consists of 
approximately 20-25 

-Sample representatii^enessj sample 
compares with the target audience 


A record is provided on 

TPDPM 7 9 f1 ) ri-P* 
r UnrJ d • a \ IJ OJ • 

-The mmber of students committing 
errors on: 

• •Each test item 




in: 

••Range of general ability 


• •Specific options in multiple 
choice tests 




••Range of specific abilities 
(relevant to masterj.^ of the 
criterion behavior) 


"The percentage of students in the 
total sample committing (sane 
type of) errors 




••Range of entering proficiency 
on the' criterion behavior 


-The nwnber and percentage of 
students committing errors on: 

••All items related to each 
sub- criterion behavior 






••All items related to the 
complete criterion behavior 



ERIC 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


• 

A record of the percentage of students in the 
sample who make errors on each part or problem 
t'n tfte progruju* 


WHAT YOU WILL 
WORK FROM 


(1) Sample of approximately 25 students 

(2) The instructional program (revised on the basts 
of informal tryouts) 


WHAT YOU WILL 
DO 


(1) Administer the program 

(2) Score and identify errors 

(3) Compute percentage of errors made on Individual 
problems and on the whole program 


FORMS YOU WILL 
USE 


FORM J. 2(1) for recording errors on the Instructional 
program. 
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[ 



DESCRIPTION OF Sub-STEP 



J. 2. 2 



INPUT 



ACTION 



Sample of students 
selected for 
develomental try out 
+ 

instructional program 
(revised during 
informal tryout) 



IV 



OUTPUT 



Administer program 
+ 

observe errors as they 
occur (for procedural 
program) ^ OR 
score errors at end 
of administration (for 
paper-and-pencil ^ 
-programs ) 



Identification of: 

-Portions of program 
on which errors aire 
made (on FORM J.2(l)) 



vi 



CRITERIA FOR 
IDENTIFYING INPUTS 


Job Aid Contends 
ACTION TO BE TAKEN STANDARD FOR OUTPUTS 


FORMS TO USE 






-MATRIX: 
Adequacy of 
program 

administration and 
identification of 
errors .... 105 


FORM J.2(l) 

SUMMARY OF 
PROCEDURES . . \0k 



Required MateriaJs 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Revi sed 

instructional program 


J.l 






FORM J.2(l) 
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Sub- STEP I J, 2. 2 I 



INPUT 



Sample of students 
selected for 
developmental tryout 
+ 

instructional progrim 
(revised during 
informal tryout) 

iv 



JOB DIAGRAM 



ACTION 



Administer prograun 
+ 

observe errors as they 
occur (for procedural 
program) , OR 
score errors at end 
of administration (for 

paper-and-pencil 
proj^rams ) v_ 



OUTPUT 



Identification of: 

-Portions of program on 
which errors are made 
(on FORM J-2(l)) 



Individual 
program problems 



t . a 



Score for whole 
tryout sample 
and compute percentage 
making errors 
on FORM J, 2(1) 



IDENTIFICATION OF 



Percentage of 
students committinq 

an error on 
each program problem 



M I .a 



All program problems 
related to each 
sub-criterion behavior 



Score for whole 
tryout sample 
and compute percentage 
making errors 
on FORM J-2(l) 



i i .b 



Identification of 
percentage of 
students committing 

errors on 
all program problems 
relating to each 
sub-criterion behavior 

i ii .b 



All program problems 
related to the 
CRITERION behavior 



I .c 



Score for whole 
tryout sample 
and compute percentage 
making errors 
on FORM J.2(l) 



I I .c 



I dent i !^icat ion of 
percentage of 
students committing 

errors on 
all program problems 
relating to the 
CRITERION behavior 



I I .c 
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JOB PROCEDURES 





page 


SUMMARY OF PROCEDURES 


]0k 


Assessing adequacy of 
Identif feat ion of errors 
committed on program 


105 
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J. 2. 2 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED IN ADMINISTERING 
INSTRUCTIONAL PROGRAM AND IDENTIFYING PROGRAM PROBLEMS 
ON WHICH ERRORS ARE MADE 



ADMINISTER PROGRAM 

3. Administer 
Instructional 
program to enti re 
tryout sample; 

• -Slmul taneously 
to all , If the 
program is on 
paper-and-penci 1 

• • One-by-one i f 
the program 
involves a set 
of procedures 



n 

OBSERVE AND/OR 
SCORE ERRORS 

Score errors for 
each (paper-and- 
pencl 1 ) program 
problem after 
program completion 

For "procedural" 
programs, observe 
practice and make 
a record of 
problems on which 
errors are made 

Make a record of 
program problems 
or program parts 
on which errors 
are made (on 
FORM J. 2(0) 



#3 

IDENTIFY 
H»ROFICIE.MCY 

a. Compute percentages 
total sample makes 
errors on 

••Each program 
problem 

b. Identify percent-- 
agei on those 
program problems 
which provide no^ 
assistance and 
are a test of the 
sub-cri terion 
behavior 

c. Identify percen- 
ages on those 
program problems 
which provide no 
assistance and 
are a test of the 
CRITERION behavior 



FORM J. 2(1) 



M <t I* IT M 



» r< a a 
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J. 2.2 

STANDARDS 
MATRIX 



ASSESSING THE ADEQUACY 
WITH WHICH PROGRAM ERRORS ARE IDENTIFIED 



PROPERTJES 



COMPLETENESS 
OF RECORD . 



CRITERIA 



On FORM J. 2(1) 
there is a record indicating 
the percentage of students 

""Making errors on each program 
problem or program task 

•Making errors on problem parts 
(for multipart problems) 

•Making errors by selecting 
specific wrong options fon 
multiple choice program problems) 



DIAGNOSTIC 
CAPABILITY 



On FORM J, 2(1) 
there are identifying marks 
which indicate 

* Which program problems provide 
no assistance and are therefore 
tests of each sub-criterion 
behavior 

•Which program problems provide 
no assistance and are therefore 
parts of the CRITERION behavior 
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ITEM #'S (N 


Option^|ndorsed| 






6 


•6 


6 


Total Number 

of People 
Making Errors 


% of Group 
Making Errors 


Performance 
Requirements 


INPUT recall 


INPUT transfer 


ACTION recall 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


--yln identification of hau) much of the criterion 
behavior the sample of students has learned as a 
result of taking the instructional program. 

^^An interpretation of the sources of or reasons for 
errors cormitted on the program. 


WHAT YOU WILL 
WORK FROM 


(1) Sample of students who have completed the 
Instructional program. 

(2) Criterion and sub-criterion tests. 


WHAT YOU WILL 
DO 


(1) Administer tests as an AFTER test. 

(2) Score tests and Identify errors. 
\S} Identity causes or errors. 


FORMS YOU WILL 
USE 


FORM J»2(l) for recording errors on tests 
FORM J. 2(2) for identifying error patterns 
FORM J*2(3) for identifying error patterns 



o 
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DESCRIPTION OF Sub-STEP S J. 2. 3 



(NPUT 

Tryout sample 
which has completed 
instructional program 

£Uld 

tests 

vii 



ACTION 

Administer tests 
as an "after" test 
and 
score tests 
and 

interpret errors 

viii 



OUTPUT 

Identification of 
how much of criterion 

and sub- criterion 
behaviors the sample 
now possesses 
and 

Interpretation of source 
of errors committed ix^ 



Job Aid Contents 



CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS FORMS TO USE 



-MATRIX: 


Assessing proficiency 


. Ill 


-MATRIX: Adequacy 


FORM J. 2(1) 




"MATRIX: 


GROUP results 


. 117 


of interpretation 


FORM J. 2(2) 




-MATRIX: 


INDIVIDUAL 


results • . . 


. 125 


of errors on 




-MATRIX: 




. 151 


"after" 


FORM J. 2(3) 




-MATRIX: 


Assessing " 


sequencing" . 


. . 155 








SUMMARY OF 
PROCEDURES . 


. . 162 



Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Criterion and 
sub-criterion tests 


J. 2.1 






FORM J.2(l) 

FORM J. 2(2) FORM J. 2(3) 
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Sub-STEP 



0.2.3 



INPUT 



Tryout sample 
which has completed 
instructional program 
and 
tests 

vii 



JOB DIAGRAM 



ACTtON 



Administer tests 
as an "after" test 
and 
score tests 
and 

interpret errors 

viii 



OUTPUT 



Identification of 
how much of criterion 
and sub-criterion 
behaviors the sample 
now possesses 
and 

interpretation of source 
of errors committed^ 



GROUP error resul ts: 


-Total test 


scores 


-f^ECALL and 


TRANSFER 


scores 




-Scores on 


INDIVIDUAL 


test Items 






vi 1 .3 



Assess 
GROUP proficiency 
and identify 
types of errors 



VI I 



-Amount of degree of 
GROUP proficiency 

"Identification of 
type and frequency 
of errors 



tx.a 



Error results for 
INDIVIDUAL students: 

■Pattern of errors: 

• •Mismatchi ng of 
classes of INPUTS 
and ACTIONS 

V i i .b 



Assess 
INDIVIDUAL proficiency 
and identify 
types of 
learning fai lures 
to account 
for errors 

viii.b 



-Identification of 
types of learning 
fai lures: 

••Discriminations 

• •General izations 

• •Associations 



I x,b 



Error results: 
during DELAYED testing 



vii -c 



Error results: 

for two or more 
criterion behaviors 



vi 1 .d 



Ac>sess 
GROUP proficiency 
and identify 
types of errors 
on a delayed basis 



VI 



Assess 
GROUP proficiency 

as a means of 
assessing adequacy 
of sequenci ng 
decl s ions 



vlil.d 



-Assessment of 
RETENTION problems 



IX. c 



-Assessment of adequacy 
of SEQUENCING 
dec! s ions 



ix.d 
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BACKGROUND INFORMATION 





FIVE TYPES OF DIAGNOSIS 


page 


A. 


ASSESSING PROFICIENCY 


112-n6 


B. 


INTERPRETING ERRORS ON THE BASIS 
OF GROUP RESULTS ON INDIVIDUAL 
TEST ITEMS 


117-I2'» 


C. 


INTERPRETING ERRORS ON THE BASIS 
OF RESULTS OF INDIVIDUAL STUDENTS 
ON MULTIPLE TEST ITEMS 


125-150 


D. 


USE OF "DELAYED" TESTING 


151-15't 


E. 


ASSESSING SEQUENCING DECISIONS 


155-158 









o 
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A. ASSESSING PROFICIENCY 



■" 


( 

page 


Purposes in comparing results on 
tests administered "before" and 
"after" administration of 
i nstructional program 


113 


How to use FORM J. 2 (2) for 
recor^ding error patterns 




Purposes in recording "by items," 
"by groups of items," and "by 
students" 


115 


Two methods for interpreting errors: 
by results on individual items and 
by patterns on groups of items 


116 
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PURPOSES IN COHPARING RESULTS ON TESTS 
ADMINISTERED "BEFORE" AND "AFTER" STUDENTS 
TAKE THE INSTRUCTIONAL PROGRAM 



PURPOSES 


ASSESSING THE 
EFFECTIVENESS OF THE 
INSTRUCTIONAL PROGRAM 


IDENTIFYING WHICH 
STUDENT RESULTS 
TO USE AS A BASIS FOR 
PROGRAM REVISION 


CRITERIA 


'•'Without 'before" results to be 
used for baseline data^ it is not 
possible to assess the adequaay 
of the instructional program in 
producing efficiency : 

• •If, for example, proficiency 
were high to begin with, high 
proficiency on the after test 
would not be as meaningful as 
it would be if there were low 
proficiency to begin with 

to identify students with low 
entering proficiency and to 
determine what the program does 
for them 


Students who teated lew *^efore'* 
taking the progixm and who te<^t 
law (i.e.^ make more errors) 
''after** taking the program can 
prooide the sample of results 
which can be interpreted and 
used as a basis for program 
revision 



J. 2. 3 
DECISION 
MATRIX 



TYPE OF 
ENTR I ES 



HOW TO USE -ORM J. 2 (2) TO RECORD ERRORS 
ON TESTS ADMINISTERED "BEFORE" AND "AFTER" 
STUDENTS TAKE THE INSTRUCTIONAL PROGRAM 



INDIVIDUAL 
entries 



1. Use each raw to record the 
errors each student makes 
on the test. . 

2. Use each column to record the 
item(s) on which error ($) 

ie (cere) made. 

8. Use the swmary box to record 
the nimber of students making 
predetermined percentage of 
errors : 

••20% or more is recormended 



"TOTAL" 
entries 



Use the bottom rou) to record 
the percentacfe of students 
making an error for each 
test item. 

Use the ^'RECALL" column to 
record the total number of 
errors each student made on 
all RECALL items. * 

Use the ''TRANSFER" column to 
record the total number of 
errors each stvident made on 
all TRANSFER items. * 

Use the ''TOTAL" column to add 
the number of RECALL and 
TRANSFER errors for each 
student. 

Use the % column to express for 
each student the percentage of 
errors he made on all items. \ 



*lndividual I terns should be coded by an R (RECALL) or by a T (TRANSFER) (See example below) 

FORM J. 2(2) 
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R 


R 


T 
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1 


T 


i cnt rr««tMr nma vton f 
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J. 2. 3 

IDENTIFICATION 
MATRIX 



PURPOSES IN RECORDING ERRORS: 
BY ITEMS, BY GROUPS OF ITEMS, AND BY STUDENTS 



METHODS OF 
TOTALING 
ERRORS 



By 

INDIVIDUAL 
STUDENTS 



By 
GROUPS 
OF STUDENTS 



By 
GROUPS 
OF ITEMS 



CRITERIA 



A. 



Suming evvovs fov 
each student (in a 
rou) permits: 

**fhe assessment 
of the degree of 
proficiency (or 
lack of it) he 
has attained on 
the basis of 
taking the 
instructional 
program 

**The identifica^ 
tion of which 
students to 
select for 
further analysis 
of error patterns 



Swwring errors for 
each test item (in 
a column) permits 
a determination of 
what percentage of 
students failed 
that particular 
item (This may 
point to need for 
program revision^ 
particularly if it 
is a RECALL item^ 
i.e. 3 something 
covered in the 
instructional 
program) 

The number of 
students exceeding 
a predetermined 
error rate 
identifies the 
degree of need of 
program revision 
results 



Sunrring errors by 
RECALL items only 
and 

By TRANSFER items 
only permits an 
identifioatiofn of 
how to program has 
fai led: 

• *0n transfer only 

• •On recall and 

transfer both 
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J, 2. 3 

IDENTIFICATION 
MATRIX 



TWO METHODS FOR ANALYZING ERRORS 
AS A BASIS FOR DIAGNOSING PROBLEMS 



METHODS 



CRITERIA 



Analysis of group results on 
••INDIVIDUAL items 
••GROUPS of items 



-Identification of individual 
test items failed by a 
predetermined percentage of 
students 

"Identification of specific types 
of item failed by a 
predetermined percentage of 
students: 

**RECALL items 

••TRANSFER items 

"Identification of total error 
scores 



Analysis of results of 
1 nd ividual students on 

••MULTIPLE items 



-Analysis of the pattern of errors 
made by individual students 

••Does the same student make 
the same or different type 
of error on: 

/Multiple items 

/RECALL/TRANSFER items 
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Items 
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j TRANSFER 


No. of 
Items 
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n USE OF GROUP RESULTS 
^ • TO ASSESS PROFICIENCY 







Three types of GROUP measures 
useful in assessing proficiency and 
in identifying the need for revision 


118 


How to interpret GROUP error scores 
on total test 


119 


How to interpret GROUP error scores 
on RECALL and TRANSFER Items 


120 


How to deal with GROUP error scores 
on individual test items 


121 


Analyzing GROUP error scores on 
individual test items 


122, 123 







117 



J. 2. 3 

IDENTIFICATION 
MATRIX 



THREE TYPES OF "GROUP" MEASURES USEFUL IN ASSESSING 
PROFiCtENCY AND IN IDENTIFYING THE NEED FOR PROGRAM REVISION 



TYPES OF 

"GROUP" 

MEASURES 



Group error score on: 
••TOTAL test 



Group error score on: 
••All RECALL Items 
••All TRANSFER items 



Group error score on: 
'•^INDIVIDUAL test items 



CRITERIA 



Computation of a (froup 



errov score involves: 

-Smming the nmhev of 
errors for each 
student on all teat 
items covering a 
sub-cr*iterion 
behavior or a 
criterion behavior 

-Expressing (for each 
student) the number 
of errors he made as 
a percentage of the 
total number of 
errors possible 
(total number of test 
it ens) 

-Counting the total 
number of students 
exceeding a 
predetermined error 
percentage and 
expressing that 
number as a percent- 
age of the total 
number of students 
taking the test 



Computation of a group 



error score involves: 

-Surming the number of 
errors for each 
student on all RECALL 
items separately ami 
on all^ TRANSFEB items 
separately 

-Expressing for each 
student tlio number of 
RECALL (TRA'^SFER) 
errors he made as a 
percentage of the 
total number of 
RECALL (TRAJ^SFER) 
errors possible 

-Countijig the total 
number of students 
exceeding a 
predetermined error 
percentage and 
expressing that 
number as a percent- 
age of the total 
nionber of students 
taking the test: two 
percentages are foiinci 
one for RECALL anc*. 
one 'for TRAllC^FER 



Computation of a group 
error score involves 

"Summing' the number oj 
students for each 
test item 

-Expressing the sum 
(of students) as a 
percentage of 
students making 
errors on that item 



SEE PAGE 


119 j 


120 


121 











e.g. , 

-S^udeiit ^1 - Mode 
wio^ on _5 t2^t itm& 
out 0^ a total 0({ SO 
ItQJU, ion. an eMo^ 
EXAMPLES 'latd of, lO'd 

-Student H - h\adt 
QA/io^ on 10 <Xm6, 
{^ofl an oMofi ^atz of^ 
10% 

-Stude.yvt ^n - Mode X 

qAAO^ |(04 an WlOK 

-IS oat oi SO 6tudQ.nt£» 
tuid an QAAofi natt OfJ 
205; OK mo/ie 

-ZQi 0|5 tiie, 6tJudz^tU 
exceeded an zAAofi 
fiato. oi 20% 



e.g. , 

-Student ^1 - Made no 
QAAofU on the, 20 
RECALL itm&, ioK an 
WioK KdtQ, od 0%, and 
S enA0fi6 on tixt SO 
TRAMS FER Itomi^, icK an 
QAJioK fiato. oi 17% 

'Stixdtnt ^n had an 
QAAOK KoXz on RECALL 
-ItemA oi Xi and on 
TRAWSFER Um6 of^ Vi 

-5 out oi SO &tudznt6 
[o^ lOi) had a 
RECALL dAAon, njxtz Oj{ 
20% 

-10 iitndznXJti oat oi SO 
loA. 40%) had a 
TRAWSFER wioK Koto, 
oi 20% OK mOKQ. 



e.g. , 

'On to^t Item ^1 , 20 
6tnd^nts oat oi SO 
6tadznX/>, OK 40% oi 
the lokolt gKoap, made 

QAKOKi^ 



118 



DECISION 
MATRIX 



DETERMINING HOW TO INTERPRET 
GROUP ERROR SCORES ON TOTAL TEST 



CONDITIONS 



ACTION 
TO TAKE 



A HIGH percentage of students 
score below a predetermined 
error rate 
on TOTAL test* 



-Consider the program in 
reasonably good shape 

-Fevieu) the RECALL and TRANSFER 
scores to determine whether the 
errors which do occur divide up 
between the two types of items: 

'•Evenly^ OH 

'* Unevenly 

(See page 120 for next steps to 
take) 

'Review the error rates for 
individual test items to determine 
if there are unusually high error 
rates for particular items 
(See page, 121) 



A HIGH percentage of students 
score above a predetermined 
error rate 
on TOTAL test* 



"Definitely consider the program 
in need of revision 

•Review the RECALL and TRANSFER 
scores to determine the extent 
of difference between the two 
types (as a basis for interpret- 
ing possible program weaknesses) 
(See page 120} 



'Review the error rates for 
individual test items to determine 
if there are unusually high error 
rates for particular items 
(See page 121) 



*Which predetermined error rate is selected 
a convenient and often used rate. 



IS arbitrary; a 20 percent rate is 



e.g. t 

•In a txyout 6mplz of^ 40 6tud^nt!i, 
90% oi tkz 4>iudzyvt6 had eA/ioK 
Kot^ oi 19% on. 12^6 

-TranMiQJi kzvIqs^ ol H,^ult6 would 
fae condiictzd to Idznil^y ^pzcL^lc 
typz^ 0^ IzoAning ^aLluA.e. i^h^ch 
could 4CAue 06 i/ie faoiXA ^on. 
modo^t pA.ogKam n,z\)i^lOYi 



e.g. , 

-In a t/iuout 6cijnplz of^ 40 6tudtyitSi, 
40% of^ tht htudzYVt^ had zkkok 
KoteA) 0/5 10% ofi moKz 

and oi Kz^ult6 ^oh, indLvidual 
^tadentA [on maZtiple. toM AjteM ) 
appzafi in onAeA Ojii a meajiA 
idznttiylng the. Iza/inlng iaXluKOAi 
vohA^ch hav2. occuAJitd 
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J. 2. 3 
DECISION 
MATRIX 



DETERMINING HOW TO INTERPRET GROUP ERROR SCORCS 
ON RECALL AND ON TRANSFER TEST ITEMS 



CONDITIONS 



Error rates 
by types of items: 

RECALL: Low 

TRANSFER; Low 



Error rates 
by types of i terns : 

RECALL: High 

TRANSFER: High 



Error rates 
by types of items: 

RECALL: Low 

TRANSFER: Htgh 



ACTION 
TO TAKE 



The group error rate 
for all test items is 
loWy and further 
analysis reveals an 
even distribution of 
lou error rates for 
RECALL and for 
TRANSFER items: 

^No diagnostic clues 
are provided by these 
results 

'-It may still be 
useful to look at 
individual test items 
to detemine whether 
there are items on 
which unusually high 
error rates occur 
(as a basis for 
search for specific 
sections of the 
program to account 
for this result) 



An equally high RECALL 
and TRANSFER rate is 
not diagnostic of 
specific types of 
problems: 

-Ccmsider the entire 
program in need of 
revision 

--Bo further analyses: 

••Review g^oup_ 
results on 
individual items 
(See next page) 

••Review individual 
results on multiple 
items (See next 
section on page 
125) 



RECALL rates are low 
and TRANSFER rates are 
high : 

-Consider ^^generaliza- 
tion" as the possible 
learning failure j 
AND 

-Do further analyses: 

••Review group 
results on 
individual items 
(See next page) 

••Revie^j) individual 
results on multiph 
items (See next 
section on page 
125) 



EXAMPLES 


e.g., 

RECALL Resul ts 

tO% 0(5 tht 4>iixdtYit 
6amplz mad^ lt66 than 
20% znKofi/i on /Ltcjolt 
AJtem6 

TRANSFER Results 


e-g. , 

RECALL Results 

25% 0^ thz htxxdtnt 
sampZ^ madt mdkt than 
10% QAAou on fitaalt 

TRANSFER Results 


e.g. , 

RECALL Results 

51 OjJ the. 4^tadzYVt 
sample, made, 1^6 than 
20% on KttaZt 

Jitmi^ 

TRANSFER Results 


15% tht 6tudtnt 
6amptz marfe lz66 than 
20i QAJiou on iJianSiiQA. 


40% oj{ tha 6tudcnt 
6ampZ^ madt mon,t than 
20% QAAou on tAan6f^e/i 


351 tht 6tixdtnt 
6amplz madt mo/ie tlian 
20% Q/oiou on t/ioMfieA. 


itm6 


itmt> 


Itm^ 
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J. 2.3 
DECISION 
MATRIX 



DETERMINING HOW TO INTERPRET 
GROUP ERROR SCORES ON INDIVIDUAL TEST ITEMS 



CONDITIONS 



LOW grcjp error rate 
on I nd i V i dua I tes t i tems 



ACTION 
TO TAKE 



-Do not do any fwther analysis 



H1C!-! grou p error rate 
on individual test 1 tems 



'Group error rates per se on a 
single test item are not 
diagnostic 

'When revising the program^ review 
the portion of the program 
relating to a specific test item 
for possible deficienoies 

-Obtain additional information 
aboJAt the types of wrong answers 
produffed as a means of gaining 
diagrjpstia infomation (See 
next page) 



EXAMPLES 



e.g. , 

-On tQ.Kt Um #?3. n Gi tkz 



e.g. 



-On tOMt Itzm ^15, 34% Of^ tiia 
6tadtnt ^asnplt made QAAon/s 

poAXicaloA. tifpoM oi QAAo^ mado. 



ERIC 
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J. 2. 3 
IDENTIFICATION 
MATRIX 



TYPES OF ERRORS WHICH CAN BE IDENTIFIED 
WHEN ANALYZING GROUP RESULTS ON INDIVIDUAL ITEMS 



TYPE or 

ERROR 



OMISSION 
of a correct ACTION 



"Either on a multiple choice item 
or on a production item, students 
may omit an answer 

'FOm J. 2(1) may be used to 
indicate how many students 
omitted ati answer for a 
particular test item 

(Time students for test item //I 
below) 



WHICH Incorrect ACTION 
was taken 



-Either on a multiple choice item 
or on a production item^ students, 
may give an Z'tLcorrect answer: 

**Mai-f take an ACTION which is 
associated with a class of 
INPUTS other than the one used 
in the test item 

"FOm J. 2(1) may be used to 
identify which wrong ACJTION (out 
of the two or more — for the two 
or more classes of IWVTS 
involved in the problem) 

•*How3 bj Cj <ij e can be used to 
designate wrong ACTIONS iiy 
iii^ etc. 

(In the example below, 32 
students took ACTION i, which 
was wrong, and 13 students 
took ACTION if, which was 
wrong) 



EXAMPLE 



TAS:< ANALYSIS DIAGRAM 



FORM J. 2(0 



Mc«1l Ib.t 



Mean Ik.H 



Tr«nitar le.ill 



LESSON 1 


1 (SUB) 


ITEM #'S (J 




2 


3 


4 


S 


Optioni^dortMl 


1. 

omissions 


5 










b. 

\ 


32 










c. 

i I 


)3 










d. 

AMI 












e. 
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*Thi8 is correct ACTION (because test item used a "2c" INPUT) 
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J. 2. 3 
DECISION 
MATRIX 



DETERMINING HOW TO ANALYZE AND INTERPRET 
GROUP ERROR SCORES ON INDIVIDUAL ITEMS 



GOAL 



ACTION 
TO TAKE 



To RECORD 
GROUP error results 
for individual test items 



-For test items (identified on 
FORM J.2C2)) on which there are 
high group error rates^ select 
the test papers (or answer 
sheets) of all those students 
who made errors on a particular 
item 

FORM J, 2(1) J code rows 
Cy dj and e for types of errors: 

Omission 



> Wrong ACTION: 
etc. 



^J ^^J ^^^^ vv^ 



-Record on FOEUd J. 2(1) the 
frequency with which each type 
of error is made 



To INTERPRET 
GROUP error results 
for individual test items 



"Interpretation based on individual 
items is undependable 

HOWEVER 

'Identify which type of error 
predominates: 

* *An omission 

** Which incorrect ACTION is 
most frequently taken 

'This can form tJie basis for 
analyzing program materials to 
account for this particular type 
of error pattern (See next page) 



FORM J. 2(2) 



FORM J. 2(1) 




J. 2. 3 

DECISION 
MATRIX 



CONDITIONS 



ACTION 
TO TAKE 



DETERMINING HOW TO INTERPRET 
GROUP RESULTS ON INDIVIDUAL TEST ITEMS 



OMISSION 
of response 



INTERPRET as: 

-A failure to acquire 
relevant ckills 
(either dieorirrtina- 
tionSj generaliza- 
tions^ or assocria^ 
tions) 

-Which skill is not 
acquired is 
indeterminate 



ROUGHLY EQUAL 
DISTRIBUTION 
of types of 

wrong ACTIONS 



INTERPRET as: 

-An unsystematic 
learning failure (of 
relevant skills) 



-i/hich skill is not 
acquired is 
indeterminate 



UNEQUAL DISTRIBUTIONS 
of types of 
wrong ACTIONS 

(one type of Incorrect 
ACTION predominates) 



INTERPRET as: 

-A systematic learning 
failure (of relevant 
silfiZtc) 



-One or more of the 
relevant skills has 
been learned 
incorrectly 



EXAMPLE 




The test item involves classifying an example of a 
lever as a 1st, 2nd, or 3rd class lever. The example 
used is a 1st class lever. 



e.g. , 

'0(^ tho^z making znxoHM^ 
omiJXzd making any 

-Wo lQ.a/i>tLng koi 
occuAAzd 



e.g. , 

•0^ tho^t making znAoK^ 
on thU iXm, 33S 
omitttd ay\^iotnA,ng , 
331 zndoM^zd 2nd cJioAA, 
and 33% zndouzd ind 

•Thz ^aiZuAZ id 
un6((6tematic 
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e.g., 

-Of^ tkohz making 
QAAoA^ on thi& itm, 
80% mdoM^zd 2nd 

-T/ie f^aibjJiz i6 
6i46tmatic 



NOTE: 



As an introduation to proaedures 
to follcM in interpreting learner 
difficulties which result in 
errors on tests^ REVIEW PAGES 
73-90 IN SECTION "F" OF THE 
HANDBOOK. 



I NTERPRETI NG ERROR -PATTERNS 
FOR INDIVIDUAL STUDENTS ^ 
ON MULTIPLE TEST ITEMS 





page 


f n rce way ^ inuiwipit' lc^l i Lciiib 
make interpretation of error 
patterns , possible 


128, 129 


Fr^rrnc t*^ iic^ Jn an^?lV7lnn 
r 0 11 lib LU use III aiicJiy^-iiiy 

error patterns on multiple 
test items 


130, 131 


P rnrpdurfx; fnr fi 1 1 ina OUt . 
scoring, and analyzing FORMS 
J. 2(0 and J. 2(3) ^or individual 
students 


132, 133 


Identifying error patterns 


13A, 135 


FOLDOUT with subject matter 
examples 




Selecting students whose test 
scores should be analyzed for 
error patterns 


136 


Guidelines for estimating confidence 
with which interpretations can be 
made 


137 


Illustrative error patterns and 
interpretations 


138-1^7 


Recommended procedures for 
identifying error patterns 


l/*8 



The procedures renommp.nded in this section represent an innovative and, 
as of March 1972, an unvalidaied approach for diagnosing learning failures. 
Moreover, because the procedures are highly complicated, the reader is 
advised to: review this section, attempt the associated WORKBOOK 
exercises, and, should experience difficulties, shrug them off. 



ERIC 
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FOLD OUT THIS PAGE 
AND REFER TO IT WHILE 
YOU READ PAGES ]2Q-\hO. 



EXAMPLE 



INPUT 



Recal 1 la. i 






Recall la.M 






Transfer la.iil 






Recall Ib.i 





Recall Ib.i 



Transfer Ib.lil 



ACTION 



ACTION 2.i 




OUTPUT 



Recall Ic.l 






Recall Ic.m" 






Transfer Ic.iii 





ACTION 2.iH 



ERIC 
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J,2.3 

MATRIX 



THREE WAYS MULTIPLE TEST ITEMS ASSIST IN DIAGNOSING 
THF TYPF OF LEARNING FAILURE WHICH HAS RESULTED IN TEST ERRORS 



ANALYSES 
MULTIPLE ITEMS 
ALLOW 



CRITERIA 



Sampl i ng 
of INPUT classes 



hhiltiple test ite'us 
permit to he 
represented in the 
test: 

-Tioo or more of the 
INPUT classes which 
hove to be 
discriminated 

and/or 

'Tdo or more members 
belonging to the same 
INPUT class which 
require generaliza- 
tion 



Assessing the 
CONSISTENCY 
of errors 



Comparison of 
RECALL and TRANSFER 



Use of multiple test 
items allows the 
assessment of the 
consistency with which 

-Is the same or a 
different wrong 
ACTION taken to two 
or more INPUTS 
belonging to the 
same class? 

- Js the same or a 
different wrong 
action taken to 
INPUTS belonging to 
all the INPUT classes 
or to sane of the 
TfJPUT classes? 



Use of rm^ltiple test 
items J some RECALL 
items and sane 
TRANSFER J permits a 
comparison of: 

-Errors occurring in 
one but not the 
other type of test 
item 

'Errors occurring in 
both types of test 
items 



EXAMPLES 

(See Illustration 
on oppos i te page 
for notation) 



■ERJC- 



Multiple test Items 
permit the sampling of: 

-The three classes of 
INPUTS: (la, lb, and 
Ic): 

••Three test items 
can provide one 
example from each 
cl ass 

-The multiple members 
wi thin each class : 
(i , i i , and i i i ) : 

••Nine test items can 
provide one example 
for each member 
represented in the 
diagram 



Multiple test items 
permit the assessment 
of whether errors are 
cons istent or 
i nconsistent: 

-Is ACTION (2.li) 
I ncorrectly taken: 

• -For just (ia. 1) , or 

••For (la. i i) and 
( 1 a . i i i ) as we 1 1 

-Is ACTION (2.ii) 
I ncorrectly taken: 

• •For just class la) , 
or 

• -For both class (la) 
and (Ic) 



Multiple test items 
pc r?T5 i t a cGmpariScn Oi 
RECALL and TRANSFER 
i terns : 

-Is ACTION (2.iii) 
incorrectly taken: 

••For just TRANSFER 
(la , i i i ) , or 

••For both RECALL 
(la.i) and TRANSFER 
(la.iil) 



0.2.3 

IDENTIFICATION 
MATRIX 



FORflS TO USE IN ANALYZING 
PATTERNS OF ERRORS ON MULTIPLE ITEMS 
(ON THIS AND ON OPPOSITE PAGE) 



FORMS 



FORM J. 2 (2) 



FORM J. 2(1) 



CRITERIA 



-From FORM J.' 2 (2) students with 
hicjh error scores are selected 
to oanprise the scunpte for 
pattern analysis 



-Each student has a FOnU J. 2(1) 
filled out 

-His responses to each teat item 
are recorded on the form 

-Rows a-e <xre used to record the 
foltowhxg error information:'^ 

a: ACTION for class la taken 

b: ACTION for class lb taken 

c: ACTION for class Ic taken 

d: ACTION for class Id taJ<en 

e Omissions 

-Templates ax*e used to identify 
types of errors made (See page 132 
for procedures) : 

**For each INPUT class 

**For RECALL and TRANSFER 



^Record which wrong ACTION was taken; e^g., if the INPUT presented in the test item 
was a class lb INPUT and student produced the ACTIOM which is supposed to be 
associated with class Ic, an X would be put In row c; an X in row d would mean 
that the ACTION for class Id was produced. 
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F«rmJ7|;) 

USSOf^ C 



Ij (SVSf CRfllRfON BE HA ViOH L 



resrenRORs 



ITEMS 



STUg^NTS 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


1? 


13 


14 


15 


16 


17 


18 


19 


20 


21 


22 


23 


24 


25 


26 


27 




























































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































of S*.»ctl"^U 





























































IflANSFEK 


TOTAL 






No. J« 
hnnv 


No. ut 
Itrnj 













































































































































































































































Form J:?t1> 

USSON 



iSUB} CRfTERtON BBHAVfOR I 



SUMA ■/ ' QF£Ff^ORSONCRI fffHON r£S T 
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ITEM JVS 0 
Optiont Endoncd 


1 


2 


3 


4 


5 


6 


7 


"a 


9 


10 


11 


12^ 


13 


14 


15 


16 


17 


18 


1 ■ 

1 


70 


2\ 


22 


23 


24 


25 


a 


















































b 




















































c 




















































d 




















































e 
















































i 










































































































fVifonTUtioc 
Requirements 




3 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 


16 


17 


!« 


20 


21 


22 


2Z 


24 


25 






















































INPUT tf«nj<M 




















































ACTION raM" 




















































ACTION ttJf*!**' 




































1 
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IDENTIFICATION 
MATRIX 



FORMS TO USE IN ANALYZING 
PATTERNS OF ERRORS ON MULTIPLE ITEMS 
(ON THIS AND ON OPPOSITE PAGE) 



FORMS 



FORM J -2 (3) 



TABLES 



CRITERIA 



"FoT each student the types of 
wvong ACTIONS taken which were 
identified on F0R14 J. 2(1} are 
recorded on FOBld J* 2 (3)^ 
separate ly : 

Q)**By each INPUT class 

0- *By each ACTION class taken 

(i)**By RECALL items 

TRANSFER items 

-The types of errors are 
simnarised and categorised (7) 

"An error pattern is identified 
for the student (7) 

(See page 133 for procedures) 



-The student ^s error pattern is 
compared with patterns in the 
tables provided in this section (J) 
and the type of learning failure 
identified 



(See page 233 for procedures) 



ERLC 
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FORM J. 2 (3) 



TABLE 



© 



® 



C\AUOf AfllOM TAKfM 



' 0) ',c rJnift^r toi 
J Ai^oelat ion 



Cenera I ! /at ion 



1 Dl scr inlnAt inn 
J Assoc 1 at lort 



Di serininatioo ■ 
} AssocTitlnn 



' Ol'scrinlriAtron 
' Association 
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Sub-STEP 
J. 2. 3 



PROCEDURES FOR COMPLETING FORM J. 2(1) FOR EACH STUDENT 



!• Identify the INPUT clast sampled In each tost 
Item wUh a Roman number; Class i, I i ,111, 

}V» 8tC, 

2. Use fO¥* for ACTION classes corresponding to 
INPUT classes (l'IV« etc*}* pius « row for 
oialsslons 



3« Record (from ortgiaal test psper) the 
wrong ACTION taken <lf any) or omission 
(Ho entry In a colunn rneans that Item 
was' correct) 

4. Identify by checking the appropriate category 
whether a test item requires INPUT ReCALL or 
INPUT TRANSFER 

(Proceed below to procedures for preparing 
a template for scoring types of errors) 



FORM J. 2(1) 



EXAMPLE 













II 






II 


»»» 












1 


I 


1 




1 


n 


B 










1 














f 




1 




































1 














































MCltOv lit 




















































»CflO« IV 


























































































































































































































» 






* 


( 






11 
















« 




a 




M 


« 








/ 




/ 




/ 








/ 


















































/ 







































































































































PROCEDURES FOR PREPARING A SCORJNG TEMPLATE 



Create a template for each of the IMPUT 
classes Involved In the criterion or sub- 
criterion behavior: 

..The tenplate should expose aU^ the 
test Items and the possible answers 
for each class of INPUTS 

••In the example below all the answers 
to Items for INPUT Class t are 
expdsad 

..On one Item fl a Class Ml actloo was 
taken; and on /6» a Class It action 
was taken 



2. For each INPUT class, add up separately for 
RECACrTtems and for TRAHSfER Items: 

-The number of correct answers 

-The number of Incorrect ACT I QMS taken by 
type (I.e., I, II. HI. etc) 

-The number of omissions 

«.ln the example below these results are 
shown: 

/For RECALL items (no correct answers, 
one II wrong ACTION, one Ml wrong 
ACTION) 

/For TRANSFER Items (no correct answers, 
one II wrong ACTION, no I M wrong 
ACTION) 



FORM J. 2(1) AS A TEMPLATE (EXAMPLE) 





















KTIM 1 
tfllM It 
KTIM III 
KflH IV 


1 




1 


III 

^•l-'.-'.-yjt 


■ 


















X'. 














































































































































































/ 




































































































i 
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PROCEDURES FOR RECORDING TEMPLATE RESULTS ON FORM J. 2 (3) 



Record the sums obtained from 
the template analysis on 
FORM J ,2 (3): 

••For each INPUT class 

••Separately by RECALL, 
arid TRANSFER items 



© 



Summarize the results of the 
analyses for each INPUT class 
by the" fol lowing categories: 

••Number correct 

• •Number omi tted 

••Number of cons is tent 
mismatches of INPUTS 
and ACTIONS 

••Number of incons istent 
mismatches of INPUTS 
and ACTIONS 



(30 Obtain an error pattern by 

adding up the number of classes 
resulting in each category 



••SEE NEXT PAGE FOR PROCEDURES 



FORM J. 2(3) 



EXAMPLE {RECALL portion only) 



Template analysis 
is recorded here 

Read record 
as follows: 

-For INPUT 
Class I: 

• • Zero 

correct ACTIONS 

• *Cne Class II 

ACTION 

••One Class III 
ACTION 

-Etc, 



CLASS OF ACTrON TAKiN 



0 


1 


t 










0 


0 




2 




cofrtct 

2 












2 


0 






2 












CLASS 


0 


0 


2 










2 










0 
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Record in(2)i3 
simmarized and 
categorized here in(^ 

-For INPUT Class I: 

**Zero correct ACTIONS 

" 'Zero omissions 

• 'Tu)o (inconsistent) 

I (iisr fiat oiling s 

A portion is filled out 
inQ)by adding up 
results in (2j) e,g, ^ no 
INPUT classes resulted 
in missions 



PROCEDURES FOR COMPARING OBTAINED PATTERN WITH PATTERNS IN TABLES 



Using both the RECALL pattern 
and the TRANSFER pattern you 
have obtained (the example 
above has only RECALL), find 
a table in thi s^ section of the 
HANDBOOK wi th which it 
corresponds. 



If there is a table to match 
the one you have obtained, 
read off the diagnosis. 

Repeat all these procedures 
for al 1 students whose results 
are to be analyzed in detail. 



TABLE 



EXAMPLE 



CCoff^ct . ACTION I j ACTIONS MIS 

ACTION omittad 1 I awMtianrly ii 



MISMATCHED 
i neon intent I V 



RECALL 







RECAU 


RECALL 







DiPriCULTIES 



CentdintficitMr 



3 Of scr iffll nat ion 
a Association 
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J. 2.3 
DECISION 
MATRIX 



DETCRMINING HOW TO DEVELOP A RECALL OR TRANSFER PATTERN 
BASED ON THE SUMMARY OF NUMBER OF ITEMS 
FALLING INTO EACH CATEGORY (BY CLASS OF INPUTS) 



CONDITIONS 



ALL items for a given 
INPUT class fall 
Into ONE eel 1 : 

• 'Al 1 correct, OR 

• 'Al 1 omi tted, OR 

"Al 1 consistently 
mismatched, OR 

• 'Al 1 inconsistently 
m ismatched 



ACTION 
TO TAKE 



-Categorize that class 



in keeping with the 
100% endorsement 



I tems for a given 
INPUT class are 
approximately equal ly 
divided across cells 



-Do not categorize that 
class 

-TJtat automatica'ily 
makes it impossih le 
for the subsequent 
totaling of tlie number 
of classes falling 
into any category from 
beiyig ''all classes'' 

-The RECALL pattern or 
the TRANSFER pattern 
wouldy there fore^ 
involve either "no " 
classes 
classes 



A MAJORITY 
(or near majority) 
of items for a given 
INPUT class fal 1 
into ONE cell 



'Categorize that class 
in keeping with the 
majority endorsement 

SEE NOTE BELOW 



*The larger the number of items there are covering an INPUT class AND the more of 
them falling into a given cell, the more dependable the finding and the more 
dependable the diagnosis based on it* 



JNPUT 
CLASS 



EXAMPLES 



ERIC 



SUMMARY OF NO. OF ITEMS 

conntttnt inconiitttni 
correct om.jifon mnmatch.r»fl 



'On aJU iJouA ta^t Item 
coveA^ig Iitpat Cto4 

omiitzd an a>i6toe.A 
- INPUT Cla^6 n should'^ 

"onUted" 



SUMMARY OF NO. OF ITEMS 



correct omillic^ 



consistent inconsistent 
misrrwtching 



1 



1 



1 



1 



'ThiPUT Cla^6 mtxld 
not bp ocut.zqoHA.ZQji 



SUMMARY OF NO. OF ITEMS! 

consistent mconitittnt 
correct om.snon mismttchmfl 





1 


2 


1 



-A maK majoKLtij o,^ 
AXojn^ fiOZhi jintjo tliz 

ing" cxitZQOMj 
'Ccutegonlze WPUT 

^*aon6Z&te^itly 
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J. 2. 3 
EXAMPLES 



EXA/1PLES ILLUSTRATING HOW TO FILL IN 
A RECALL OR TRANSFER PATTERN 







SUMMARY OF #1 
NO. OF ITEMS 


DISCUSSION 




oorract 


omiSSion 


[:on|iit«nt inconiittant 
miimtitchino 


-Each of the four '^categories*' 
19 app 1 1 CdD 1 e on 1 y lo bume v,^ 
INPUT classes 


1 


1 


1 


1 


2 




tl 

INPUT 
CLASS 


0 


0 




' T 




** LorrecL is appticaoie oniy 
to Class III (b) 


5 


0 


0 


0 




** umission la appiicaDis on 1 y 
to Class IV 0 




1 


3 


1 


0 




* * Lons 1 s ten L is app 1 1 cao i e 
only to Class 1 1 (5) 


V 


2 


0 


0 


3 ^ 




••"Inconsistent'' is applicable 
to Classes 1 and V (t) 




COrrtct 


omiliion 


coniiitant mconiiltant 
miim«tchtng 




no claiwi 










FILL j;/ 

£7?/?0;? PATTEHN 
HERE 




^ lome clawi 


/ 


/ 


/ 


/ 




•11 cittses 






























corrtct 


#2 

SUMMARY OF 

NO OF news 

contKtant incomntint 
m„m.tch.r^ 


DISCUSSION 

-The category ''correct'* Is ^ 
applicable to no^ classes w 


\ 


0 


1 


h 


0 




-The category "omission'* Is 
applicable to no classes (?) 


W 


0 


2 


3 


0 




-The category "Inconsistent" is 
applicable to no classes(h) 


INPUT 11, 
CLASS 


1 


1 


3 


0 




-The category "consistent" is 
applicable to al 1 classes Q 


IV 


1 


1 


2 


1 






V 


0 


0 


3 


2 








correct 


sx 

omission 


:onti«ent inconintent 
mismatching 




no ciasws 


/ 


r 

/ 










tome cItssAi 














all ciaiws 
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J. 2,3 

DECISION 
MATRIX 



HOW TO SELECT STUDENTS WHOSE PERFORMANCE 
ON MULTIPLE TEST ITEMS SHOULD BE ANALYZED 



STUDENT 
PERFOR^tANCE 
ON TOTAL TEST 



I .e. 



Students with 

LOW error rates 

on TOTAL test: 

less than a 20^ error rate 



Students wi th 
HIGH error rates 
on TOTAL test: 

I.e., more than a 20^ error rate 



ACTION 
TO TAKE 



•Do rot inalude the results of 
these students in the detailed 
analysis of error patterns 



"Consider the results of these 
students for inclusion in the 
detailed analysis of error 
patterns 

Select from among these students 
those with error rates in the 
range of 30%^60% (if available) 

'Exclude students with error rates 
of 70% or higher 

"Select a sofnple of approximately 
tsn to ticenty students for this 
analysis 



RATIONALE 



ERIC 



Low frequency of errors makes It 
difficult to detect patterns of 
errors 
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•A small sample of students is 
recommended (i.e., n = 10-20) 
because this type of analysis 
is time consuming 

-Error rates in excess of 60^^-70?; 
are probably indicative of no 
learning having occurred rather 
than of specific failures to be 
di agnosed 



J. 2. 3 
DECISION 
MATRIX 



VARIABLES TO CONSIDER IN ESTIMATING 
THE CONFIDENCE WITH WHICH INTERPRETATIONS CAN BE MADE 



CONDITIONS 



ACTION 
TO TAKE 



■The more test items there are 
reflecting the same error pattern 

AND 

-The more students there are 
exhibiting the same error pattern 



WTERPRET 

-The more teat items there are 
reflecting the jsame error pattern^ 
the loW'Zr the probability that 
the results were obtained by 
chance 

-In combination with the above 
finding^ the higher the percentage 
of students exh%b^ting the same 
error pattpm^ the more dependable 
the interpretation 

'The key variable ^ however^ is the 
nionber of test items 



-The fewer test items there are 
reflecting the same error pattern 
AND 

-The fewer students there are 
exhibiting the same error pattern 



INTERPRET 

^The fewer test items there are 
reflecting the same error pattern 
the higher the probability that 
the results were obtained by 
chance 

-Jn combination with the above 
finding^ the lower the percentage 
of students exhibiting the same 
error pattern^ the less dependable 
the interpretation 



EXAMPLES 




-There are twe 1 ve test items, each 
item having a different example 
of a lever (four examples for 
each of the three classes of 
levers) 

-All four items from one of the 
classes are given the same wrong 
classification 

-The probability of this occurring 
by chance is relatively low 

-Interpretation is eas ier to make 



'1 



-There are three test items, each 
item having a different example 
of a lever (one example for each 
of the three classes of levers) 



-One item from a class is given 
a wrong classification 

-The probability of this occurring 
by chance is relatively hlgh^ 

-Interpretation is harder to make 



AN EXPLAMATION OF WHAT THE DIAGNOSTIC TABLE ENTRIES SIGNIFY 



Each of the numbered statements below refers to a oomparably numbered 
section of the illustrative diagnostic table below or to the task analysis 
table on the opposite page, 

1. Rc^s In the table Indicate how many classes were treated in particular 
ways by the student; all the JNPUT classes, some of them> or all of 
them. For the example on the opposite page, that means, how were each 
of the three classes treated by the student. All means that three 
classes were treated In a particular way; some means that one or two 
were treated J., a particular way; and n£ means that none of the classes 
were treated in a particular way. 

2. uulumns identify the ways in which tNPUT classes were treated: 
'' Correct ACTIONS couH have been taken 

••ACTIONS could have been omi tted 

••INPUTS and ACTIONS could have been mismatched: 

/ Consistently (i.e., multiple items from the same JNPUT class 
led to the same wrong ACTION, belonging to another class) 

/ inconsistently (i.e., multiple Items from the same INPUT class 
led to varied wrong ACTIONS) 

3- The different pattern applying to RECALL items and to TRANSFER items 
are indicated by entried labeled RECALL or TRANSFER 

r^ultiple entries for either RECALL or for TRANSFER items are possible 
indicating that classes can be treated in a variety of ways (correct, 
Incorrect, and with omission) 

5. Diagnoses appear in the last two columns. 



ILLUSTRATIVE DIAGNOSTIC TABLE* 

















(D 




Correct 




ACTION 




ACTIONS MISMATCHED 






DIFFICULTIES 






ACTION 




omitted 




consistently 


inconsistenlly 






Contraindicated 




Indicated 




INPUTS 












































no 

classes 






TRANSFER 










1 


Discrimination 




1 General i zation 




some 
ciassQs 
















2 


Association 




2 




all 


(D 

RECALlT 














3 






3 





















i 



*The illustrative table should be read as follows: on RECALL items all IvJPUT classes 
were treated correctly (i.e.j the correct ACTIONS were produced); on TRANSFER items: 
no INPUT classes resulted in oniasion of an answcrj and all IiJPUT classes resulted in 
inconsistent misnatching of HIPUTS and ACTI0113. 

IlJTERPRETATIOiJ: The nost likely difficulty indicated is ■\jQneraliv.ationJ' In 

descending order of probability y the difficulties contraindicated 



are ^'discrimination^' and "association. 

m 
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TABLES 



The next several pages present 
di agnostic tables interpreting 
error patterns v/hich are more 
easily interpreted than other 
possible patterns. Whenever 
the patterns you encounter in 
your work are not illustrated 
here and it is unclear what 
learning difficulties produced 
test errors, you may have to 
resort to the administration of 
diagnostic tests and interview 
probes (See Sub-STEP J. 2.^4). 



FOLD OUT THIS PAGE FOR 
SUBJECT MATTER EXAf-IPLES 
WHICH WILL BE USED TO 
ILLUSTRATE ERROR PATTERNS 



ILLUSTRATIVE SUBJECT MAUER AND TEST ITEMS 



CRITERION BEHAVIOR: 
CRiTERION TEST: 



To classify examples of levers belonging to each of 
the three classes of lovers. 

Twelve test items comprise the criterion test: 

••Four items apiece test for each class of levers 
(INPUTS) 

••Of the four items for each class: 

/Tv%/o are RECALL Items (contalninq examples used 
i n i nstruct Ion) 

/Two are TRANSFER items (containino examples not 
used in instruction) 



EXAMPLES USED IN EACH 
OF TV/ELVE TEST ITEMS: 

CJass I 



RECALL 




Class il 



Class in 



RECALL 









2 





10 



TRANSFER 














7 




n 



TRANSFER 





)) 8 





12 
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SOME DIAGNOSTIC TABLES 



1^3 



o 

ERIC 





Correct 
ACTION 




INPUTS 




no 

classes 


RECALL 

TRANSFER 




some 
classes 






all 

classes 







ACTION 
omitted 



ACTIONS MISMATCHED 
consistenily inconsiftentty 





DIFFICULTIES 




Contraindicated 




Indicated 



RECALL 
TRANSFER 











RECALL 
TRANSF^ER 





^ Discrimination 
2 Generalization 



INPUTS 



no 

clas'es 



«>nie 
classes 



all 

classes 



INPUTS 



classes 



some 

CloSSCS 



all 

clashes 



IN'PUTS 



no 

classes 



same 
classes 



all 

classes 



INPUTS 



no 

classes 



some 
classes 



all 

classes 



INPUTS 
no 

classes 

some 
classes 

all 

classes 



'^Recall and transfer items are mism atched in the same wa^j. 



RECALL 
TRANSFER 



RECALL 
TRANSFER 













RECALL 
TRANS ^^R 



1 

2 

3 Association 



**ReoaLl and transfer items are nisn atahed in different ways. 



'^^ECALL 
TRANSFER 







RECALL 
TRANSFER 


RECALL 
TRANSFER 







Sys temat ic 
association 



1 

2 Discrimination 

3 General i zat Ion 



1 
2 

3 Discriminations 
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EXAMPLES ILLUSTRATING PATTERNS ON OPPOSITE PAGE* 



EXAMPLES 



PATTERN 



RESULTS 

a. All classes of levers have 
been m! sm atched , for both 
RECALL and TRANSFER Items 

b. The mismatching was consistent : 

(1) Each of the classes of 
levers (I, III) led to 
a different (wrong) ACTION 
(label i ng) 

PLUS 

(2) Each of the examples within 
a class led to the same 
(wrong) ACTION (label ing) : 

• • 1 , 2 , 3 , k we re 
classified as 111 

• • 5 , 6 , 7 , 8 we re 
classified as I 

••9, 10, n , 12 were 
classified as { I 



INTERPRETATION 

(1) Since each of the quartets 
of items represents a 
different class and all 
quartets were classified 
in different (wrong) ways, 
discrimination is NOT the 
problem 

(2) Since members of each 
quartet represent the same 
class and each of the 
members is labeled the 
same (wrong) wai/j 
generalization is NOT the 
problem 

The problem is one of 
sifstematicallij wrong 
associations 



PATTERN 

#2 



All classes of levers have 
been mi sm atched > for both 
RECALL and TRANSFER items 

The mismatching was 
i neon s i stent: 

(1) Some of the classes were 
treated the same way other 
classes were: 

••I and 11 were treated 
the same way; OR 

••I and III were treated 
the same way; OR 

**ll and III were treated 
the same way 

(2) SOME of the examples 
within a class were treated 
with a different (wrong) 
response : 

••1 and 3 were classified 
1 1 1 

••2 and k were classified 

n 



(1) classes which should have 
been treated differently 
were treated alike 

(2) Examples within classes 
which should have been 
treated alike were treated 

•differently 

The problem is probably one of 
discrimination and generaliza- 
tion 



PATTERN 

#3 



a. Some classes were treated 
cor recti y; some i ncorrect ly^ 

b. Incorrect treatment was 
consistent for some classes 
and inconsistent for others 



Problem may be that of 
di scrimination 



♦Numbers used in the example refer to the test items on foldout page 1^0 



1^5 



8 

















Correct 
ACTION 




ACTION 
omitted 




ACTIONS MISMATCHED 
coniistentiv inconsistently 




DIFFICULTIES 
Contraindicated 1 ndicated 




INPUTS 












no 

classes 






TRANSFER 










1 


Discrimination 




1 General Ization 




some 
classes 
















2 


Association 




2 




alt 

classes 


RECALL 










TRANSFER 




3 






3 




INPUTS 












no 

classes 






TRANSFER 










1 


Discrimination 




f Generalization 




some 
classes 


TRANSFER 














2 


Association 




2 




ail 

classes 


RECALL 








TRANSFER 






«J 






3 




INPUTS 
no 

classes 
















TRANSFER 










1 


Discrimination 




^ General izat ion 




some 
classes 


TRANSFER 














2 


Associat ion 




2 




all 


RECALL 










TRANSFER 










3 




INPUTS 
no 

classes 




























1 


Discrimi nation 




1 General ization 




some 
classes 
















2 


Association 








all 

classes 


RECALL 




TRANSFER 










3 






3 




INPUTS 
no 

classes 






























1 


Discrimination 




^ General i zation 




some 
classes 


TRANSFER 




TRANSFER 










2 


Associat ion 




2 




all 

classes 


RECALL 














3 






3 




INPUTS 














no 

classes 
















1 






1 




some 
classes 
















2 






2 




all 

classes 
















3 






3 
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EXAMPLES ILLUSTRATING PATTERNS ON OPPOSITE PAGE* 



EXAMPLES 



PATTERN 



RESULTS 

On RECALL items, all classes 
and members within classes 
have been correctly treated 

Mismatching of INPUTS and 
ACTIONS occurs for TRANSFER 
items only 

b.l Consistent 

b.2 Inconsistent 

In some cases TRANSFER items 
are correct (for some classes) 
or are omitted (for some 
cl asses) 



INTERPRETATION 

Since all RECALL items are 
covrecty correct 
discriminations and 
associations ha^e occurred 

Only TRANSFER items are 
incorrect — suggesting that 
generalization to unencountered 
examples has failed 



1 
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J. 2. 3 
DECISION 
MATRIX 



DETERMINING HOW TO USE RAW DATA OF 
PATTERN ANALYSIS TO GAIN FURTHER DIAGNOSTIC INFORMATION 



TYPE OF 
INFORMATION 

1 111 w i\i in 1 1 wii 

AVAILABLE 


Data in matrix 

nn FORM J it"^) 

regarding INPUT classes 


Data in matrix 
on FORM J. 2 (3) 
regarding ACTION classes 




For each student 
in the diagnostic sample: 


For eaJi, student 
in the diagnostic sample: 


ACTION 
TO TAKE 


^nhti^yrii^ -{w^Yi ^/7/»?i TPJ'PII^ 

where the errors pile up; i.e.j 
in ACT ion Class ly II, III, 
or V, or in an omission 


where the errors pile up; i.e., 
in INPUT Class I, II, III. IV, or 
Vj or in an omission 




-Observe uhether for other ItlPVT 
classes the pile up is: 


-Observe whether for other ACTION 
classes the pile up is: 




**T^ie srvne 


••The some 




• 'Different 


• •Different 




-Use this as raw data for 
identifying the confusions which 
exist 






COrlPAEE THE RESULTS WITH THOSE OF 
OTHER STUDEHTS W THE DIAGNOSTIC 
SAl'fPLE 









Look for the distribution 
of errors across a row/^ 


Look 
of 


for the distribution 
errors down a column* 


EXAMPLES 








CLASS OF ACTION TAKEN 






CLASS OF ACTION TAKEN 








1 


II 


HI 


IV 


V 


Omisiioni 




1 


II 


III 


IV 


V 


OmitdOns 








correct 
















correct 


1 


















correct 
















correct 












INPUT 
CLASSES 


II 
















II 

INPUT 
CLASSES 




0 












ill 


0 




correct 
1 


3 


2 






III 




k 


correct 












IV 








correct 








IV 




2 




correct 










V 










correct 






V 




0 






correct 






*For 

has 


illustrative purposes 

bepn f M 1 pH in 


, only one 
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J. 2. 3 


TWO VARIABLES USEFUL FOR IDENTIFYING DIFFICULTIES 


IDENTIFICATION 
MATRIX 












VARIABLES 


t 

EVENNESS 
of distribution of errors 

••For INPUTS (across ACTIONS) 

••For ACTIONS (across INPUTS) 


■ DEGREE OF OVERLAP 
across classes 


CRITERIA 


-For each INPUT class is there: 

*'An even distribution across 
wrong ACTIONS 

OR 


-For two or more classes of 
INPUTS is there: 

•*Qoerlap in the type of 
distribution across ACTIONS 




•*An uneven distribution ft.e.^ 
pile up) across wong ACTIONS 


••Non-overlap in the type of 
distribution across ACTIONS 




-For each ACTION is there: 


-For tDo or more ACTIONS' is there: 




''An even distribution across 
INPUT classes 

' 'An uneven distribution across 
INPUT classes 


• 'Overlap in the type of 
distribution across INPUT 
classes 

* *Nan-overlap in the type of 

distribution across INPUT 
classes 





For INPUTS: 


-The more overlap in an even 


INTERPRETATION 


-Uneven distribution for an INPUT 
class means that there is a 
systematic error for that class; 


distribution of errors there Is, 
the more likely is there a 
failure to discriminate among 
cl asses 




-Even distributions for an INPUT 
class means non-systematic errors 
for that class 


-The less overlap there is In an 
uneven distribution, the more 
likely is the failure to be one 




For ACTIONS: 


of association 




-Uneven distribution for an ACTION 






means a systematic error 






-Even distributions for an ACTiON 






means a non-sV^temat fc error 




O 







ERJC )/,9 



J. 2. 3 



EXAMPLES ILLUSTRATING VARYING DEGREES OF OVERLAP 
(EXAMPLES ARE HYPOTHETICAL TO STRESS PARTICULAR PATTERN) 



EXAMPLES 





CLASS OF ACTION TAKEN 
1 li lil IV V omiMiOnt 




#1 

-The distribution of errors, 
omissions, and correct iters 
is even: 

••For INPUTS across ACTIONS 

••For ACTIONS across INPUTS 

-Tlie overlap among 1 NPUT *c lasses 
and among ACTIONS is high 

TJw most probable failure is: 
di s crimina tions 


ti t 

1 

II 

INPUT 
CLASSES 

IV 
V 


correct 
1 


1 


1 


0 


0 


1 




1 


correct 
0 


1 


1 


1 


0 


0 


1 


correct 
1 


] 


0 


] 


I 


0 


1 


correct 
0 


1 


\ 


I 


I 


0 


1 


correct 
0 


] 










CLASS OF ACTION TAKEN 
1 II III IV V omissions 




n 

-The distribution of errors, 
omissions, and correct items 
is relatively uneven: 

••For INPUTS across ACTIONS 

••For ACTIONS across INPUTS 

-The overlap among INPUT dresses 
and among ACTIONS is low 

Tke most probable failure is: 
associations 


n 

II 

INPUT 
CLASSES 

IV 
V 


correct 
0 


3 


0 


0 


0 


1 




0 


correct 
1 


3 


0 


0 


0 


k 


0 


correct 

0 


0 


0 


0 


0 


0 


0 


correct 
2 


0 


2 


0 


0 


0 


0 


correct 

5 


0 










CLASS OF ACTION TAKEN 
1 II III IV V omissions 




-The distribution of errors, 
omissions, and correct items 
is neither even nor uneven 

-The overlap among INPUT classes 
and among ACTIONS is neither 
high nor low 

The pattern is not readily 
interpretal)te* However^ acme 
of the specific confusions 
(mismatchinns) m*e identified 


#3 

1 

II 

INPUT ,„ 
CLASSES 

tv 

V 


correct 
1 


0 


3 


0 


0 


0 




2 


correct 

0 


2 


0 


0 


0 


1 


1 


0 


1 


1 


0 


0 


0 


0 


correct 

2 


2 


0 


0 


0 


1 


1 


correct 
1 


1 
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i£$SON f (SUB) CmriRION BEHAVIOR ^ 



STUDENT ANALYSIS 



RECALL items 



INPUT 
CLASSES 



CLASS OF ACTION TAKEN 
" IV V omissioni 



SUMMARY OF 
NO. OF ITEMS 



1 


correct 












II 




correct 










III 






correct 








IV 








correct 






V 1 










correct 





corrtci omission 



INPUT x\\ 
CLASS 



consistent inconsistenl 
mismatchirg 



no ciasiss 



RECALL 
PATTERN 



consistent inconsistent 
correct omiuion misntetching 



TRANSFER items 



INPUT 
CLASSES 



CLASS OF ACTION TAKEN 





1 


II 


III 


IV 


V 


omissions 




correct 












II 




correct 










III 






correct 








IV 








correct 






V 










correct 

1 





CLASS 



SUMMARY OF 
NO. OF fTEMS 



correct omission 



consistent inconsistent 
mismatching 



TRANSFER 
PATTERN 



correct omjssion 



consistent inconsistent 
mismatching 
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RECALL items 



CLASS OF ACTION TAKEN 
< Hi IV V omiuions 



INPUT 
CLASSES 



correct 














correct 














correct 














correct 














correct 





INPUT ,1, 
CLASS 



SUMMARY OF 
NO. OF ITEMS 

contittint incontistfnt 
om.«.on miimttching 



RECALL 
PATTPRN 



consistent incontittvnt 
correct omiuion mitmatching 



TRANSFER items 



INPUT 
CLASSES 



CLASS OF ACTION TAKEN 



1 


II 


III 


IV 


V 


omissioni 


1 


correct 












n 




correct 










III 






correct 








IV 








correct 






V 










correct 





INPUT 
CLASS 



SUMMARY OF 
NO. OF ITEMS 

coniiiicnt inconsistent 



correct omission 



mismatching 



consistent inconsistent 
correct om.ss.on ^i^^.ch.nfl 



no classes 



TRANSFER 
PATTERN wme classes 



all classes 
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USE OF ''DELAYED" TESTING 





page 


Two times at which to administer 
tests: "immediate" and "delayed" 
bas is 


152 


In which parts of the developmental 
tryout cycle should these two types 
of testing be used? 


153 
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DIFFERENT PURPOSES IN DEVELOPMENTAL TESTING 
FOR PROFICIENCY AT DIFFERENT TIMES 



TIME OF 
TESTING 


Testing 
IMMEDIATELY AFTER 
students complete program 


Test Inq 
ON A DELAYED BASIS 
after students complete program 


CRITERIA 


^Purpose is to determine whether: 

• •Students haoe aacfuired 
sub-criterion and criterion 
behaviors 

••l^at types of failures have 
occurred (which skills are not 
adequately learned) 


-Purpose is to determine whether: 

••Students have retained 
what they had already 
acquired from the progvam 

•'Which specific skills are 
forgotten 



ERIC 



DETERMINING IN WHICH PART OF DEVELOPMENTAL CYCLE 
TO ADMINISTER TESTS ON AN IMMEDIATE AND DELAYED BASIS 



TYPE OF 
TESTING 


IMMEDIATE 


DELAYED 


ACTION 
TO TAKE 


^AdrrrLniatev an "immediate*^ 
after test on: 

•*Each and every cycle of 
developmental txyout; i.e.^ ifj 
for example^ there are three 
cycles of tryoutj "immediate** 
tests would be administered on 
all three occasions 


^Administer a '^delayed" 
after test on: 

••Later stages of the development 
tryout cycle; i.e.j ifj for 
example J there are three cycles 
of tryoutj "delayed" tests 
would be ajdministered only 
after the third cycle 

OR 

••Administer the delayed test 
after a tryout in which high 
proficiency has been 
demonstrated and consequently 
little revision mode on tJie 
program 



RATIONALE 



It is important to determine 
whether students were able to 
acqu i re what they were supposed 
Lo learn 
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If students did not in the first 
place acquire criterion behaviors, 
delayed testing w<?uld not be 
diagnostic of retenti<?n problems 
(since no retention could be 
expected to occur) 



ASSESSING ''SEQUENCING" DECISIONS 





page 


Differences between sequencing 
decisions made in Task C and in 
Task G 


156 


Two methods for assessing 
sequencing decisions regarding 
separate criterion behaviors^ 


157 


Two methods for assessing sequencing 
decisions regarding components 
within a single criterion behavior 


158 















^SECTION up to this point has been aonaemed with 
assessing stvudent profioienoy at a single criterion 
behavior (and at the sub-criterion behaviors which 
make it up). The next several pages are concerned 
with assessing proficiency at two or more critenon 
behaviors as a means of determining the adequacy of 
sequencing decisions made in TASK "C, " 
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J. 2. 3 

IDENTIFICATION 
MATRIX 



DIFFERENCES BETWEEN SEQUENCING DECISIONS 
MADE IN TASK "C" AND IN TASK "G" 



TASKS 



Sequencing decisions 
in TASK "C" 



Sequencing decisions 
in TASK "G" 



CRITERIA 



'The decision is made as to tlie 
order in which a number of 
different criterion behaviors 
should be teamed 

'The purpose of the decision is 
to sequence learning experiences 
so that prior mastery of 
criterion behaviors which can 
facilitate the learning of other 
criterion behaviors are taught 
earlier in the overall program 



-The decision is made aa to the 
order in which components, of a 
single criterion behavior should 
be learned 

•-Components ordered in various 
ways include: 

• •Sub' criterion behaviors 

••Individual skills which make up 
tJie sub-criterion behaviors 

-Trie purpose is to order or 
sequence the learning of 
components so that mas tenj of the 
total criterion behavior will be 
most efficient 





e.g. , 


e.g. , 






-Ton. tzadUng tliz concept "ijo^^ce" 








EXAMPLES 


• • FoKct 


tzadiing tho. n-ztzvawt- 




• •PKZ66U/1(1 


• •'Di6CAA)nAJiation6 




-Thz dQ,cJj>^on /ia6 io be made >cn 


• •Gznz/iatLzatioyi^ 




ToAfe "C": In lOkcut OKdoA ^houJbl 


• •A64iO.Cyicutioyi6 




thz ti)o concept fae taught 6o 
tlxcut -teo/Litcjig can be optbnLztd 


A^nvoZvzd jin tiia cowczpt 
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IDENTIFICATION 
MATRIX 



TWO POSSIBLE METHODS FOR ASSESSING THE ADEtJUACY OF 
SEQUENCING DECISIONS MADE IN TASK 't" 
(REGARDING SEPARATE CRITERION BEHAVIORS) 



METHODS 



CRITERIA 



EXPERIMENTAL* 



-During developmental try out 
(preferably late in the try out 
md revision cycle when the 
instructional program for each 
criterion behavior results in 
relatively high proficiency) the 
programs for two or more separate 
criterion behaviors can be 
administered to different groups 
of tryout students in dvfferent 
orders : 

••Group 1 learns cri terion 
behavior 4 first and B_ second 

• • Group 2 learns criterion 
behavior 5 first and second 

-The groups are then compared for 
differences in: 

••Time to complete the 
instructional program related 
to each criterion behavior 

••dumber of errors in the 
instructional program 

••Scores on proficiency tests 
(difference between ^'before** 
and **after" scores) 

-If A should precede as 
originally decided, there should 
be substantial and significant 
differences betu)een the two 
groups ' 



STATISTICAL* 



'During developmental tryout 
(preferably early in the tryout 
and revision cycle when there are 
still relatively wide differences 
among students in performance on 
criterion tests or on programs) 
groups may be composed on one of 
the following bases: 

• • Number of errors on program 
for criterion behavior A 

••Time to complete program for 
criterion behavior A 

• •'Proficiency scores on test for 

criterion behavior A 

'Two groups are foxfned: 

• •A high scoring group 
••A low scoring ' group 

on tJie variable selected above 

'Their scores for criterion 
behavior B are then compared on 
the same variable or on other 
variables 

'(An alternative to group 
comparison is to compute a 
correlation between scores on 
A and on B) 

'If A should J in facty precede By 
high achievers on A should be 
relatively higher achievers on B; 
and low achievers on A should be 
relatively lower achievers on B 



EXAMPLES 
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e.g. , 

-GKoap 1 maid txxkz thz pKog^am ow* 

• • VoKcz and 

• •?Kti>&H/lQ, 6 2.cond 

-GKoap 2 would takz tliz p^og^iamom 

• •P^C/64>uAe (jVut, and 

• • t j'tce 6Zcond 



e.g. , 

-High ackiz\)QM on tliz '^f^o^cz" t(U^t 
and lao adUzveM on tha "|{o>tce" 
tQAt should be compoAzd {^on. 
achlzvmcLnt on thz "p>te64u/Le" te^t 
("jjo^ce*' having been taught to att 

• Cont/iolting jo^ abA.litif '' I iJ 
''ijo^^ce" precede p^te^-ioAe, " 

kighs on ''^oKct'' ^hoixld be tkz 
(vcg(i6 OYi '^pfie^^oKz" 



157 



TWO POSSIBLE METHODS FOR ASSESSING THE ADEQUACY OF 
J. 2. 3 SEQUENCING DECISIONS MADE IN TASK "G" 
IDENTIFICATION (REGARDING COMPONENTS OF A SINGLE CRITERION BEHMIOR) 
MATRIX I 



METHODS 


EXPERIMENTAL 


STATISTICAL 


CRITERIA 


'During developmental try out 
(preferably late in the try out 
and revision cycle when the 
instructional program for each 
criterion behavior results in 
relatively high proficiency) the 
programs for two or more separate 
sub-criterion behaviors related 
to the same criterion behocoior 
can be administered to different 
groups of try out students in 
different orders: 

••Group 1 learns sub -criterion 
behavior 4 first and B second 

••Group 2 learns sub-criterion 
behavior B first and second 

-TJze groups are tJwn compared for 
differences in: 

••Time to complete the 
instructional program related 
to each sub- criterion behaoior 

• •limber of errors in the 
instructional program 

• •Scores on proficiency tests 
(difference between %efore^* 
and after'* scores) 

-If A should preced.e as 
originally decided^ there should 
be substantial and significant 
differerices between the two 
groups 


-During developmental tryout 
(preferably earli^ in the tryout 
and revision cycle when there are 
still relatively wide differences 
among students in perfomance on 
svb-Qriterion tests or on 
programs) groups may be composed 
of "^he following bases: 

••Number of errors on program 
for sub-criterion behaoior A 

••Time to complete program for 
sub-criterion behaoior A 

• •Proficiency scores on test for 
sub -criterion behaoior A 

-Two groups are formed: 

• •A high scoring group 

• *A low scoring group 

on the variable selected above 

-Trieir scores for criterion 
behavior B are then compared on 
the same variable or on other 
variables 

-(An alternative to group 
comparison is to compute a 
correlation between scores on 
A and on D) 

-If A should^ in fact, precede 
high achievers on A sliould be 
relatively higher achievers on B; 
and low achievers on A should be 
relatively lower achievers on B 






EXAt-lPLES 


e.g., 

T/te cAitcA^on bdiav^oA. Ajnvobj^ 

beJiav^o^ involve ^^dAAtcXion o{i 
f^oKcd" and "^t/izngtli of^ ijo/ice" 

-GKoup^ 1 ojid 2 u)ould d^{^izK in 
tilt o^doA in lolilch tiity took tlxaZ 
panJL oi tilt pMgAJxm on **doitction" 
a^id "6t/izngtli" o^ a ^oA.ct 


e.g., 

-/Ug/i and tao adiitve/u ^tga/cdinci 
"dOitction" { taught d^t] womtd 
be compoAtd ^OK poAf^onmanct 
KzgaAcUng "6t/LQ.ngtli" [taaglit 
6zcond) 

'ContAoZlinjpi {^o\ ability: T/icAe 
6h,ouZd be a ctc^ jje/tence be^oeen • 
kigki and lorn on "^dt/izngth" i^ 
tht 4eaaence -cA icLcltUxiXAMt 
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JOB PROCEDURES 





page 


Priorities in selecting types 
of analyses to perform 


160 


Conditions requiring assessing 
developer sequencing decisions 
and student delayed retention 


161 


SUMMARY OF PROCEDURES 


162 


Adequacy of procedures for 
interpreting test results 


163 
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J. 2. 3 
DECISION 
MATRIX 



PRIORITIES \H THE ANALYSIS OF TEST RESULTS 
AS A MEANS OF IDENTIFYING TYPES OF 
LEARNING FAILURES WHICH HAVE OCCURRED 



PRIORITIES 



1st 

Analysis of GROUP 
resul ts 



ACTION 
TO TAKE 



-ALWAYS start with 
GROUP results first 

•Analyr.e GHOUP results 
in the following 
order: 

(1) Total test scores 

* ' Low error scores^ 

do WT continue 

* "Relativehj high 

error scores^ 
continue 

(2) Test scores swrmed 
for aU RECALL 
items and oianmed 
for all TRANSFER 
items 

(3) GROUP error score 
for individual 
test items 



2nd 

Analys Is of resul ts 
of INDIVIDUAL students 



'Perform analysis of 
patterns on errors on 
multiple test items 
for INDIVIDUAL 
students ONLY l/UEN 
prior analysis of 
GROUP results 
indicates the 
ezristence of mac' or 
extension problems and 
which are not readily 
diagnosed by group 
results 



3rd 

Use of special 
DIAGNOSTIC TESTS 

CIMU 

INTtERVIEWS 
See Sub-STEP J.l.h) 



-Administer diagnostic 
tests (developed in 
TASK F) and conduct 
probing interviews 
ONLY WHEN the 2s t and 
2nd priority 
approaches both 
provide insuffi cien t 
diagnostic informa- 
tion: 

••The types of 
learnifig failures 
still are 
indeteminate 



RATIONALE 




-These types of 
analyses are easiest 
and least time- 
consuming to do 

-They can provide 
Information that: 

•*There are no major 
1 earn I ng prob le^"*^ 
(in which case no 
further analysis is 
needed) 

••There are major 
problems (in wh i ch 
case further 
analysis is 
indicated - See 
next column) 



•This is a time- 
consuming procedure 
and is therefore only 
undertaken when the 

•GROUP approach 
provl des I nsuf f icient 
evidence as to the 
types of learning 
f ai I ure whi ch have 
occurred 
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-Thi s , too, i s a time- 
consuming procedure 
and is therefore 
undertaken when both 
the GROUP and 
INDIVIDUAL approaches 
provide insufficient 
evidence as to ti";e 
types of learning 
fa I lure wh ich have 
occurred 



J. 2. 3 

IDENTIFICATION 
MATRIX 



IDENTIFYING CONDITIONS WHEN IT IS NECESSARY TO ASSESS 
FOR DELAYED RETENTION AND FOR SEaUENCING ADEQUACY 



TYPE OF 
ASSESSMENT 



CRITERIA 



Assessing 
DELAYED RETENTION 



COrWITIONS 

'Iflien statement of objectives 
calls for performance on a 
delaiied basiSy administer 
crzterion tests to a try out 
sample — late in the developmental 
cycle (when the program has 
already undergone one or two 
cycles of revision and. has 
resulted in inrnediate (non- 
delayed) high proficiency) 



Assessing effects 
of SEQUENCING DECISIONS 



CONDITIONS 

^It is always useful to assess 
the adequacy of sequencing 
decisions made both with respect 
to: 

• ^Separate criterion behaviors 

• •Components within a single 
criterion behavior 



ERIC 



J. 2. 3 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED IN 
ADMINISTERING, SCORING, AND INTERPRETING RESULTS ON TESTS 



#1 

a. Administer criterion 
and sub-cri terion 
tests to tryout 
sample of approxi- 
mately 25-50 students 
Immediately after 
thei r completion of 
the program 

b. Score tests and 
record on 

FORM J. 2 (2) 

c. Compute for the 
GROUP: 

••Total error scores 

••RECALL and TRANSFER 
SCO res 

• • Error scores on 
individual items 

d. Select high error 
students on specific 
test items and 
analyze their errors 
on FORM J.2(l): 

• •Omi ss ions 

• •Speci f i c wrong 
ACTIONS tdken 



n 

•If further analysis 
I s needed to i dent i fy 
types of learning 
fai lures : 

a. For a sub-sample 
of approximately 
ten high error 
students , fill 
out FORM J, 2(1) 

b. Make up a template 
to score for error 
patterns on 

mul ti pie test 
i terns wh i ch 
individual 
students exhibi t 

c. Identify the 
pattern and inter- 
pret it from 
tables in this 
section of ;he 
HANDBOOK 

'^Fill out a separate 
for'n fov each student 



#3 

Administer tests 
on a del ayed basis 
to students in 
later cycles of 
development 
tryout; assess 
re ten t i on 

Assess sequenci ng 
deci s i ons : 

• •Experimental ly 

• •Stat isti cal ly 

for two types of 
probes : 

• -Order of two or 
more cri terion 
behaviors 

•■Order of 
components of a 
single cri terion 
behav f or 



FORM J. 2(1) 



FORM J. 2 (2) 



FORM J. 2 (3) 



1 l- 



1.-4.-1 



■ \ \ :t 



-rt 



-.4- 



ft! -r- 



M ; 

TTiruTititL 




ERiC 
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J. 2. 3 

STANDARDS 
MATRIX 



ASSESSING THE ADEQUACY OF PROCEDURES 
FOR INTERPRETING TEST RESULTS'^ 



PROPERTIES 



CRITERIA 



-ERIC. 



COMPLETENESS 



EFFICIENCY 



-GROUP error scores 
are analyzed for: 

• * Total number of 

students exceeding 
a pred^termiyied 
error rate 

• ^differences betioeen 

error rates for 
RECALL and TRANSFER 
items 

Distributions of 
wrong answers 
ejidorsed for each 
item 

"Individual error 
scores on multiple 
items are analysed 
for patterns 
indicative of types 
of learning failures: 

• •Discriminations 

• * Generalizations 

• *Associati.ons 

"Delayed tests are 
aduinistered to 
assess ''re tention " . 

'Analyses are 
conducted to 
determine adequacy of 
sequencing of program 
materials 



'No more analyses 
(which are time- 
consuming) are 
performed than are 
required 

'Priorities in 
conducting analyses: 

--GROUP a>talyses 

Individual analyses 

* ^Special diagnostic 
procedures (See 
Sub-STEP J. 2. 4) 



SAMPLING ADECUACY 



For GROUP Analyses : 

'Canplete try out savple 
used on ''total" scores 

-For analysis of 
individual items only 
students who commit 
errors 

For 

Individual Analyses : 

'Approximately 10 
students makiiig a 
relatively high 
nwnber of errors 
(e.g.;, 30%-60% 
error rate) 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


An idenHfioation of the typos of learning failures. 


WHAT YOU WILL 
WORK FROM 


(1) Analyses of program errors. 

(2) Criterion test errors. 

("i^ DIaanostic tests and Interviej"* schedulps 


WHAT YOU WILL 
DO 


(1) Administer diagnostic tests Jn order to identify 
types of learning failures (when necessary; i.e., 
not determined in the previous sub-steps), 

(2) Conduct interviews for the santre purposes (when 
necessary) , 


FORMS YOU WILL 
USE 


None 



DESCRIPTION OF Sub-STEP 



J. 2. 4 



INPUT 




ACTION 




OUTPUT 




Analyses of 
program errors 
and 
test errors 


► 


Administer 
diagnostic tests and 
conduct interviews 
when necessary 




Identification of 
types of learning 
failures 








► 








X 




xi 




xii 















Job Aid Contents 



CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS FORMS TO USE 



-MATRIX: 


-MATRIX: When 




-MATRIX: 




SUMMARY OF 




Informat ion 


to do further 




Adequacy 




PROCEDURES . 


. 170 


sources to 


d 1 agnos t i c 




of procedures 








review .... 168 




169 


involved in 
doing further 
d i agnost i c 


171 







Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Interpretation of 
learning failures 


J. 2. 3 








Diagnostic tests 
and "probes" 


F.3 

















































ERIC 
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Sub- STEP } J. 2. 4 I 



JOB DIAGRAM 



INPUT 




ACTION 




OUTPUT 




Analyses of 
program errors 
and 
test errors 


► 


Administer 
diagnostic tests and 
conduct interviews 
when necessary 


► 


Identification of 
types of learn infs 
failures 
















X 




xi 




xii 















-Error rates on both: 
• -Tests 
••Programs 

are h i gh 

-Dependable diagnoses 
a1 ready obtained 

x.a 



•Error rates on both: 
• •Tests 
• •Programs 
are lov/ 



Do NOT do further 
diagnostic work 



x( . a 



Identification of 
sources of 
learning fai 1 ure 
(when it occurs) 



XI I .a 



x.b 



-Error rates on both: 
• •Tests 
• •Programs 

are high 

-Diagnostic work 
already performed 
resulted in 
undependable or 
unclear information 

x.c 



DO further 
d i agnost i c work 



XI .c 



Identification of 
sources of 
learning failure 
{when i t occurs) 



ERLC 
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JOB PROCEDURES 







Information sources to review 

and types of infc.matlon to obtain 


16S 


Determining whether to do further 
diagnosis of types of learning 
fai 1 ure 


169 


SUMMARY OF PROCEDURES 


170 


Assessing adequacy of procedures 
for conducting further diagnoses 


171 
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J. 2.4 
IDENTIFICATION 
MATRIX 



INFORMATION SOURCES TO REVIEW 
AND TYPES OF INFORMATION TO SEEK 



INFORMATION 

SOURCES 
TO REVIEW 



CRITERIA 



FORM J. 2 (2) - TEST RESULTS 
and 

PERFORMANCE on PROGRAM! 



-Proficiency on cri tenon tests 
and on sub-criterion tests: 

• •Percentage of students 
exceeding a predetermined 
error rate on tests'^ 

'Error rates on program problems: 

* •Criterion program problems 

• •Si4b- criterion program problems 

/Percentage of students 
exceeding a predetermined - 
percentage* 



DIAGNOSES 
made from FORM J. 2(3) 
and from FORM J. 2(2) 



•Difference in RECALL/ TRANSFER 
problems identified on FORI-! J. 2(2) 

-Specific learning failures 
diagnosed from patteryis of errors 
of individual students on multiple 
test items 



*A 20 percent error rate cutoff point is recommended. 



ERIC 
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J. 2. 4 
DECISION 
MATRIX 



DETERMINING WHEN TO ADMINISTER DIAGNOSTIC TESTS 
AND TO PROBE FOR ERROR SOURCES DURING INTERVIEWS 



CONDITIONS 



•On tests and^ on 
programs a high 
percentage of students 
(80^ or more) exceed- 
ing a 20% error rate 
AND 

-DEPENDABLE diagnoses 
obtained from analysis 
of GROUP and 
INDIVIDUAL results 



ACTION 
TO TAKE 



'Do l^OT do further 
diagnostic work 



•On tests and on 
programs a high 
percentage of students 

or more) exceed- 
ing a 20% error rate 
AND 

■Diagnoses based oir 
GROUP and INDIVIDUAL 
results are NOT 
sufficiently depend- 
able or clear 



-Administer diagnostic 
tests developed in 
TASK "F" (developing 
additional tests at 
this time if needed) 

-Conduct interviews 
with ^^igh error 
students: 

••He: test errors 
••He: program errors 



'On tests and on 
programs a high 
percentage of students 
(Z0% or more) makes 
fewer than 20% errors 



•Do WT do any 
diagnostic work 



ERIC 
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J. 2. 4 



ILLUSTRATION SUMMARIZING PROCEDURES FOR DETERMINING 
WHETHER TO AND FOR DOING FURTHER DIAGNOSTIC WORK 



#1 



REVIEW 

OUTPUT of Sub-STEP 
J. 2*3 for error 
rates on program 

OUTPUT of Sub-STEP 
J. 2. 3 for error 
rates on tests 



n 





IDENTIFY 


a . 


Error rates on 




program and on 




tests 


b. 


Percentage of 




students exceeding 




a predetermined 




standard 


c. 


Avai labi 1 i ty of 




unambiguous and 




dependable 




d iagnost i c 




i nformation 




al ready avai 1 able 



#3 



ADMINISTER 

a. Diagnostic tests 

b. Interviews probing 
for types of 
learning fai lure 
on : 

■ -Tests 

• -Programs 

when information 
identified in tfl 
is inadequate 



ERIC 
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J. 2. 4 

STANDARDS 
MATRIX 



ASSESSING THE ADEQUACY OF PROCEDUf^ES 
FOR DIAGNOSING SOURCES OF ERRORS 



PROPERTIES 



EFFICIENCY 



COMPLETENESS 



CRITERIA 



'Diacp^ostic tests a7id intewieios 
ewe used oyxly^ when: 

^•TJiere arc hogh error r'ates 
on program ami on tests 

AND 

• -It is unclear frori analyses 
of these errors what the 
sources of errors ore 



'Sources of errors (i^e.y whether 
there was a dis crimination j 
generalization^ association^ or 
chaining failure) for: 

• 'Each sub-criterion behavior 
covered by the program 

AND 

**The criterion behavior 
covered by the program 



ERLC 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


A revision in pca't(s) of or in all of the program* 


WHAT YOU WILL 

VYv/KN rKKypn 


(1) Identified types of learning failures 
\c. ) l06nLM 1 ea 50urc65> w • a i turcb 


WHAT YOU WILL 
DO 


(1) Identify program weaknesses 

(2) Revise program accordingly 


FORMS YOU WILL 
USE 


None 



ERLC 
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DESCRIPTION OF Sub-STEP 



J. 2. 5 





INPUT 




ACTION 




OUTPUT 




Identified sotirces 
of errors 
on program 
and on tests 


► 


Identify 
program weaknesses 
and make 
appropriate revisions 




Revision in: 
-Isolated problems 
-Series of problems 
-Total program 










► 






xiii 








-Sequences 

XV 















Job Aid Contents 

CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: Four 
degrees of program 



revision . . 
-MATRIX: 
I dent i fy ing 
weaknesses in 
program . • 



178 



190-195, 
20i»-206 



-MATRIX: Deciding 
on the degree of 
program 

revision . . . . 180 



-MATRIX: 
revise 
programs 



How to 



... 18^ 
196-201, 
207 

MATRIX: Revising 
sequences . 210-212 



-MATRIX: Adequacy 
of program 
revision .... 217 



SUMMARY OF 
PROCEDURES 



216 



Required Materials 



COMPLETED MATERIALS 



STEP 



COMPLETED FORMS 



STEP 



BLANK FORMS 



Interpretation of 
program and test 
errors 



J. 2. 3 



FORMS: 



J. 2(1) 
J. 2(2) 



J. 2. 4 



Diagnostic results 



J. 2. 4 



Sub -STEP J. 2. 5 



JOB DIAGRAM 



INPUT 



I'lentifiod sources 
of errors 
on proc,rani 
and on tests 



xiii 



ACTION 



Identify 
prorrran wenJcnesses 
and make 
ap-oropriate revisions 



xiv 



OUTPUT 



Revision in: 
-Isolated problems 
-Series of probloras 
-Total pror^ro/n 
-Sequences xv 



PaOGHAH WEAKNESSES 



LEANNESS 

• • Excess i ve 

• • I nsuf f i cient 



XI M .a 



Correct the 
degree of leanness 



x^ v»a 



RFVISIOfiS 



-Less lean 
••Mom assistance 

-Leaner 

• •More unass i sted 
oract i cc 



XV. a 




Renx3ve or 
clear up the 
con f us ions 



xl v.b 



Changes in: 
- Fo rma t 
-Content 



XV .b 



SPECIFIC SKILLS 
not adequately taught 



XI I I .c 



Less than optimal 
SEQUENCE 



XI i i .d 



Use techniques 
appropriate to 
the ski I Is 
to be taught 



X I V .c 



17^ 




■Inclusion of 

techniques omitted 

•Improvement of 
techn rques 
inadequately used 



XV .c 



Reordered sequence of: 

-Sub-cr i teri on 
behaviors 

-Criterion behaviors 



XV. d 



BACKGROUND INFORMATION 





page 


Overview of the tryout process 


176 


Identification of program weaknesses 
through direct evidence and through 
inference 


177 


Four degrees of program revision 


178, 179 


Conditions calling for the four 
degrees of program revision 


180, 181 


Priorities in deciding how much 
of the program to revise 


182 
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IDENTIFICATION 
MATRIX 



TWO SOURCES FOR IDENTIFYING PROGRAM WEAKNESSES 



SOURCES 


DIRECT EVIDENCE 


INFERENCE* 


CRITERIA 


-Student reports in diagnostic 
interviews directly identify: 

• •Vocabulary sentence 

difficulties 

• •Difficulties understanding 

instructions or a statement 
of objectives 

• *Exarrples of problem features 

that caused them difficulties 


-Student behavior on tasks: 

••Does something counter to 
instructions provided 

••Pedis to do what instructions 
require 

-Results on tests: 

••Nienber of errors 

••Types of wrong answers 




'Alternating program sequences 
produce differential performance 
results 

-Errors on specific program 
problems: 


••Patterns of errors 

••Errors on problems linked to 
specific component skills 




* •Individual problems 






••i4 se-ries of problems 





*The burden of Identifying program weakness usually lies most on inference from 
program and test results. 



EXAMPLES 



e.g., 6tudznt ^zpontii o/( 

^zahi/ie/i -0^' a p^OQ^m p^ovidz 

e.g., i,^ a compa/uuon KOMU^t^ 

adrnhvUtoAzd in d^^oA^rvt 
ondzM pfiodixcLQM dlf^loAzntiaZ 

zvidzncz 0^ i^Jhcut thz optimum 
^eQuence i& [and convoASoMj 
lohcut >c4 a poo^ sequence) 



e.g., a 6pzci^ic mong andmK 
p^oduczd btj a ZnAQZ 
pe/iczntagz of^ tiiz t/imvut 
hamptQ. 4aggeA^6 tliz po66i- 
bititii oi tliQ. pKOQfwm luiving 
pKodaczd tixz mAjiConczption 

z.g., a high pcAczntagz aAAon^ 
on p^tzpa/icutonif p^og^am 
p^obtQjn6 and on cAyitoAion 
pn.obZm6 6agge^t^ a total 
iYistnjuLctionaZ sequence 
iyisafif^icizntJbj cxiQjd to 
p/vodact wioK-(^KZZ pQA{^0HmancQ. 
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J. 2. 5 
(OENTIFICATIOU 
MATRIX 



FOUR POSSIBLE AMOUNTS OF PROGRAM REVISION 
(On this and on opposite page) 



POSSIBLE AMOUNTS 
OF 

PROGRAM CHANGE 



] 

Changes in 
tSOLATED 
(individual) p rob 1 ems 

throughout the 
instructional program 



CRITERIA 



-Changes are made in individual 
progran problems (or individual 
program tasks) 

-Changes node on each prohlen are 
unr elated, to changes made in 
other progran problens 



in 



Changes in 
a SEQUENTIAL SERIES of 

(several) problems 
the instructional program 



'Changes are made in a nmher of 
ser*ially ordered program problems 
(hut the nimber is less than that 
involved in the whole program) 

'Changes maie in each problem 
are related to the changes made 
in other problens in the series 
(because performance on one 
problem affects performance an 
the next, etc. ) 





e.g. 






On a pncqKm Involving MS pKa<M,ct pKobtm6 ddoZhu] loitli a 
cnlt^/Uon bekavloK, c/ionge-A ma// be made In: 




-T/ie f^oZlof^^ing JUolcutzd pn^oblQm: 


-T/te {)0(I.OiOinQ ^(>aIu of) pKobZm^'* 




••72, 29, 47, ^9, 103, U6, Uc. 


••12, 13, J4, IS, 16, and 17 

•*103, 104, 105, 106, 107, 108, 
and 109 




e.g. 






On a pnogKm Involving p^jacticQ, oi 7S ^6-tep6 lnvolvQ.d in a 
p^occAn/icit tcUik, changed tnatj be marfe in pKaatAco. invotvinn: 




-5-tep4 5, 26, 32, 6S, etc. 


-StQ.p6 S, 6, 7, 8, md 9 
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J-2,5 
IDENTIFICATION 
MATRIX 



FOUR POSSIBLE AMOUNTS OF PROGRAM REVISION 
(On this and on opposite page) 



POSSIBLE AMOUNTS 
OF 

PROGRAM CHANGE 



Changes in the 
ENTIRE PROGRAM 
related to 

•A sub-cri terl on behavior 

•A criterion behavior 



Changes in the 
SEQUENCE of programs 

••For separate sub-criterion 
behaviors (re: the same 
cri terion behavior) 

••For different criterion 
behaviors 



CRITERIA 



'Changes are made in the entire 
instructional program (but not 
necessarily on every problem or 
on every part of a practice task) 
related to one or both of the 
following : 

••i4 sub' c-rt terion behavior 

•*A criterion behavior 

'Changes made are ijxterrelated 
to one another 



'Only the sequence with which 
intact programs (or intact parts 
of programs) are administered 
is altered 



e.g.. 

On a pKoqnxm involving 125 p^acX^ct 
bdiav^oK, c/tongeA maif be made In: 

pA.obZ(m6 ^njom 1-125 {altliough not 
ma(i66aAAZy In all pnoblQ)n6) 

e.g. , 

On a pioQKm ImoJOoinQ practice 
7S 6tzp6 involved In a pKoczduKoZ 
tcu^k, diange^ maij be made 
pAncicce hwotvlng- 

'PKactict ftOK tiiz zntiAd xSe/ucei o/( 
4.teja6 (I-7S) maij be dxanQod 



e.g. , 

-Tfie sequence In tolUdi 6zpa/iaJ:(i 
p^og^aj7\6 io^ c/UtoAyLon bzhavioK A 
and cnltznion bohavlon. 8 a^e 
ojdmini^toAzd imf be ^zvoMzd 

e.g. , 

-The 6zquznc.z in loliidi p^og^am 
AZdtionA UoitJiin thd 6amt oveAaU. 
pKogKm] loK Mib' C/ujtQAion 
bdiavion^ A and S [botii ^tlout.e.d to 
thz 6 me cAltzALon bchavioA.) may 
be A.evQMzd 
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J. 2. 5 

IDENTIFICATION 
MATRIX 



IDENTIFYING CONDITIONS CALLING FOR 
VARYING DEGREES OF PROGRAf>l REVISION 
(On this and on opposite page) 





1 

CONDITIONS calling for: 


2 

CONDITIONS calling for: 


MnUUril Ur 

PROGRAM REVISION 


Changes in 
ISOLATED 
(indi V idual ) problems 

throughout the 
instructional program 


Changes fn 
a SEQUENTIAL SERIES of 
(several) problems 
in the instructional program 










-% of People: less than 20% 


-/& oy people: less than 20% 




Lil'WV rOLZ& m iUUI'& VrlCZrl cU /o 


^Error rate: more than 207^ 


r D l TC O 1 \ 

UK 1 1 tK I/\ 


• ^On the total instructional 
program 

anil 

••On the sub-criterion and 
criterion tests covering the 
program 


••On the total tnctructtonal 
program 

and 

••On the suh^criterion and 
criterion tests covering the 
program 




AND 


AND 




-% of people: nore than 20% 


of people: nore than 20% 




Error rate: more than 20% 


Error rate: nore tha>2 20% 




••On individual J isolated program 
problems or program tasks 


• •On a sequential series of 
program problems or tasks 
. and/ or 

••On test items testim for 
material novered by the series 
of problems 



ERIC 



EXA/iPLES 

••Has 100 

pxobtom^ 



^•Hcu 25 itm6 



e.g., RESULTS 



'Total to^t' 



-On pfvoQfim: 



11% 6mpl^ 
mafee6 moA.e than 
20 '6 QAKon^s 

16% 0^ 6mpl^ mafee^ 
mo^c tlian 20% CA^ou 

Mo^c tlian 20% of^ tiid 

6mptz mafce6 tM-on^s 

on pxobleM: 

6, 19, 52, 41, S2, 

dtc. 



RESULTS 



-Totat pn^ogKom*' 



9% o{) 6ct)npl(L mafee6 
wo^c than 20% 

-Total t^ti 12% 0|i' 6ai\\plQ, mafeeA 
moA.e tlian 20% qjikoka 



-On pn.oqn.mi 



m 

Mo^c tiiCLVi 20% of^ tixz 

on a 4eA/ce6 on 
]'>nohtm6: ^'/6 6, 7, 
g, 9, 10, 11, 12, and 
13 

-On toA^t: 22% o^ tlxz 6mptz mak(U> 
(iAAon^ on all toMt itmSi 
noZoutzd to cjOYitZYvt 
coveAzd in pnognojn 
pnobl07i 6-1 i 
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IDENTIFYING CONDITIONS CALLING FOR 
J. 2. 5 VARYING DEGREES OF PROGRAM REVISION 

(On this and on opposite page) 

IDENTIFICATION 

MATRIX I 



POSSIBLE AMOUNTS 
OF 

PROGRAI^ CHANGE 


3 

CONDITIONS calling for: 


k 

CONDITIONS calling for: 


Changes In the 
ENTIRE PROGRAM 
related to 

••A sub-criterion behavior 

••A criterion behavior 


Changes in the 
SEQUENCE of programs 

••For separate sub-criterion 
behaviors (re: the same 
criterion behavior) 

••For different criterion 
behav i o rs 




j ERROR RESULTS 

-% of people: More than 20% 

^Error rate: More than 20% 

••On total instructional program 
and/or 

sub-criterion tests 


EXPERmENTAL 
assessment of sequence suitability 


-Sequence A-B yields a lower 
error rate: 

••On Program A and on Program B 
and 

••On tests for A and for B 
than the sequence B-A 

STATISTICAL 
assessment of sequence suitability 


-Test results* for programs A and 
B: 

• •Low error scores on A tend to 
be low error scores on B 

••High error scores on A tend to 
be hiqh error scores on B 






EXAMPLES 

••HoA 100 

TeM: 

• •Ha6 25 Itm^ 


e.g., RESULTS 

-In tliz -Aeqaence A- 8, tliz aueAage 
CAAOA, Kotz on toMtb {^oK A and G 

-In tliz -6eqaence 8-A, Hit aue/tage 
qjVioK fiot^ on t^tSi {^OK A and B 
(004 29% 


'^.q. , RESULTS 

-In Hid -6e({aence A- 8, ihd avQAago, 
Wio/i ^dtz on toMtJii {^oH. A and 8 

-In thd sequence 8-A, tlit avoAaqa 
WioK Kotd on tZJ^tA ioft A and 8 



*Test results used in this way can be for programs under development or for 
existing programs which are thought to produce prerequisite entering behaviors. 



ERIC 
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J. 2. 5 

DECISION 
MATRIX 



DETERMINING HOW MUCH OF THE INSTRUCTIONAL PROGRAM TO REVISE 



CONDITIONS 



Error rates: 

on TESTS and/or 
on PROGRAM are: 

' Low (less than 20^^) 

•For most of the 
tryout sample (more 
than 3or^) 



ACTION 
TO TAKE 



-Do not plm extensive 
program revision 

'Identify individual 
pronircon problems or 
series of program 
problems on Dhioh 
high error rates have 
ooaiirred; revise 
these program prohl&ns 



Error rates: 

c.i TESTS and/or 
on PROGRAM are: 

* * Hlgh (more than 20^) 

• -For a large part of 
the tryout sample 
(more than 20?;) 



'Flan a revision of 
the entire program 

-Make revisions: 

**Iyi those individual 
practice problems 
or 

••In t^ie series of 
practice problems 

on which results 
indicate: 

••High error rates 
and/or 

^ •High error rates on 
test problems 
related to program 
problems 



Error rates: 

on TESTS and/or 
on PROGRAM are: 

•Substant ial ly 
di f ferent for 
alternative sequenced 
(whether over a 1 1 
error rates are high 
or low) 



'Alter the sequence of 
practice: 

• •Practice related to 
ti)o or more 
sub-criterion 
beMviors related 
to t^ie some 
criten\on behavior 

OR 

• •Practice related to 
tbX) or more 
different criterion 
helmviors 

--Hake changes in the 
programs themselves 
according to the 
adjacent columns 



ERIC 
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REVISING 
INDIVIDUAL PRACTICE PROBLEMS 





page 


Sources of information about 
weaknesses of individual practice 
problems 




Types of weaknesses in Individual 
practice problems 


185 


Revisions to make in individual 
practice problems to eliminate 
weaknesses 


186, 187 
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J. 2. 5 

IDENTIFICATION 
MATRIX 



SOURCES OF INFORMATION ABOUT WEAKNESSES 
OF INDIVIDUAL (ISOLATED) PRACTICE PROBLEMS* 



SOURCES 



CRITERIA 



"GROUP" ERROR SCORES 
on p rog r ajn 

recorded on 
FORM J, 2(1) 



-The magnitude of the 
percentage of try out 
students who made 
evvoTQ on pvogvam 
problems (or tasks) 
identifies w hioh 
progran pi'ohlern is 
in need of revision 

-Tlxe frequency of 
particular wrong 
answers or particular 
wrong procedures 
identifies the 
probable nature of 
tlie problem: 

• 'Omissions 

••Even distribution 
anonr: wrong ansvers 

••Uneven distribution 
among wrong answers 



TY""S OF 
PROGRAM PROBLEMS 

on which 
errors are made: 

identified on 
f^ORM J. 2(1) 



-Sub-criterion progran 
problems and 
criterion progran 
problems are 
identified on 
FORM J. 2(1) 

-Error rates for 
individual it ana 
obtainable from 
"GROUP" scores (See 
Qolwm to tlie left) 
can be treated 
differently for: 

• •Problems or tasks 
which are part of a 
progression (lead^ 
ing up to a 
criterion behavior) 
Aim 

••Problems which 
involve criterion 
(or Biih-criteMon) 
behavior 



INTERVIEWS 
and /or 
OBSERVATIONAL DATA 



-Interviews provide 
for student identifi- 
cation of 

diffiaiiltiaa with: 

• ' Vocabu lari!^ 
sentence a omp lexi by 
or lengtli 

• • Ins trr ic tions or 

statement of 
objectives 

• •Attention or 

observation control 
techniques 

••Content of problem 

-Observation provides 
for identification of 
sti^dent difficulties 
with: 

••Attention or 

observation control 
techniques 

••Content of problem 
or task 



^Except for those program problems v\fhich involve sub-criterion or criterion 
behavior, test data usually provide no dependable information that can be 
easily traced back to individual program problems. See the next section 
for ways to treat situations in which errors occur (either on the program 
or on the test) on measures of sub-criterion or criterion behavior. 
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J. 2. 5 

DECISION 
MATRIX 



IDENTIFYING NATURE OF WEAKNESS IN A PROGRAM PROBLEM OR TASK 
ON THE BASIS OF AVAILABLE DATA 





• • uiTi 1 b b 1 Oil u r diibwe r 


" "M pd r L 1 cu i ar wrony 


••A variety of wrong 


DATA 


or of procedure 


answer produced by 


ansv;ers produced by 


AVAILABLE 


'•Failure to attend or 
observe (as intended) 

• -Student report of 
not understanding 


many students 

••Frequent attention 
to a wrong problem 
feature 


many students 




INTERPRET as 


INTERPRET as 


INTERPRET as 




evidence of one or 


evidence of one or 


evidence of: 




more of the following : 


more of the following: 




~7ocdbulccpy or 






ACTION 


language complexity 






TO TAKE 


inappropriate for 
target audience 

^Absence of or 
incomplete: 

-•Task instructions 

• 'Techniques to 
control attent^on/ 


-Task instructions or 
attention control 
techniques are: 

• • Absent J inaomp le te 

• 'Misleading 






observation 


-Cuing provided for 


-Cuing provided for the 




'Insufficient cuvng 


the problem may be: 


problem may be 




to enable a correct 
response at that 


• •Insufficient 


insufficient at that 
point in the 




point in a 


• '!^Hs leading 


progression 




progression 







EXAMPLES 



o — 

ERIC 



e.g., 

'VocabaloAij lavzl ok 
^CK^tCKice tznoitii may 

the. ago, oft gnjodo. 
tkz Aitide^ttA Jin tkz 
tangeJ: au(icence 



o[e6/cgned to coni/ioZ 
attoyi^bion mat/ be 

coiiSilnQ student to 
atte^nd to mong 
in^omation 

-C£u^'A pKO\jidQ.d i^ltli a 
p/iobZexn maff mislead 
^tiidzYitii yinto 
p^odaclng a poJuttcuJia/t 



e.g., 

'CiUng matj have, been 
■totatZtj LOltlidAoi'M in a 
pn.obZm too ^oon 

'Pa/itiat ciUng provided 
U.e., hidAjiQ^ct 
cudJiiln^tancQ] man be 
Inadzqaoitz {\Ofi thz 
pKoblm 
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J. 2. 5 
DECISION 
MATRIX 



DETERMINING HOW TO REVISE INDIVIDUAL PROGRAM PROBLEMS OR TASKS 
TO OVERCOME IDENTIFIED WEAKNESSES 



CONDITIONS 



Problem FORilAT 
• • I nsuf f i dent 
or 

• 'M f s lead i ng 



'Simplify or atari fy 

• • Vooabulcay 

• •Sentence struoticrQ 
••Task instructions 



BEHAVIOR CONTROL 
TECHNIQUES 

Insuff ic'ent 

or 

• -Misleading 



-Simplify or clarify 

• '^Attention or 
observation control 
techniques or 
devices 



CUING 

• • Insu/f icient for 
the problem 

••Misleading 



'If no cuiiig had been 
provvded^ consider 
the fd liming 
possibilities : 

• •Changing the 

example to an 
easier one fe.gr. ^ 
one already 
practiced) and 
leaving out cuing 
again 

• 'Adding indirect or 
partial auing 

-If indirect or 
partial cuing was 
used originally J make 
it stronger 

-Do not resort to 
direct i comp lete) 
cuing unless the 
problem is one 
occurring very exrly 
in a i)rogression 



EXAMPLES 



ERLC 



e.g., 

'Cat doion ^ZYitoncz 

-ti^e liigh ^Kzqumcy 

-U4e tve/iifdajj language 
in p^ace Oj$ ILtQAoMj 
Zangu e o/l tzchnical 

hoLnguxigt 



e.g., 

•Hake, attantlon cont/wZ 
dzviczs ZjOAgoA, moKt 
pHDminQ^YVt, zoubWi to 
f^oZtoio, etc. 

'Pnovtdt ^o/i attontton 
contAjoZ voh^n -it 
x([L^hing} e.g., "Loofe 
aX. tliz dld tanaz Hit 
load Aji (^n.om tiiz 
\utcnim on tlvU IzvoaJ' 
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SEE OPPOSITE PAGE 



J. 2. 5 
EXAMPLES 



EXAMPLES ILLUSTRATING HOW WEAKNESSES OF INDIVIDUAL PROGRAM PROBLEMS 
OR TASKS CAN BE OVERCOME BY CHANGING CUING 



LEVELS 
OF CUING 



ORIGIML PROBLEM 



MORE CUING 



STILL MORE CUING 




A 



TTTi 



Tkz man Iia6 jii6t 
apptizd a {^on,c.^ to tiiz 

CjOAt. 

In lokat ciOizctioh mMt 
thz aa/iX be mov^^ng? 



Thz man ha6 ja6^ 
p^j^hzd -t/te caAt. 

In what cLUizction rrnjust 
tixz ca/it bz movA^ng? 




(T~r) 



Thz man hai ju^t 
pii6hzd tixz (ua/it. Thz 
ca/ut ma6t bz moving ^n 
thz 4a/ne d^^zction. 

In tohat dl/izction mii6t 
tliz coAt bz mo\}i.ng? 



n 



ConcZitt/iatz on making 
a donjizct loop (oi 
ijoii tivtzad tiuJi 
\oKojzctoA] , 



Concznt/iatt on making 
a loop (04 ifou tliAzad 
this f^iJbn joKojzctoA] , 
!\akz 6UAZ tixz loop i^ 
nzithzn, too long noK 
too 6hoKt, 



Conaznt/iatz on making 
a loop (oA (joa tlxKzad 
tlvu f^ihn pnjojzQyton.) . 
l\akz MOiz tliz loop dozA 
not QXdZzd tixzMz giiidz 
ma/ik&, [Guiidz maA.fi6 
pKOvidzd] 



ERIC 






i^ tivU? 



bihat cZa6^ IzvzA 
i$> tlvU? 

A substi tution of 
example makes this an 
easier problem. 
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Notz ti\at thz 
AZMi^tancz ioficz is 
bzMjQzzn tixz (iiilcAum am 
zf^ioKt ioAcz, bJhat 
c£a64 ol IzvzA ifi tkU? 



REVISING 

SEQUENTIAL SERIES OF PRACTICE PROF/ -MS 





-1 

page 


Sources of information about 
weakness of a series of practice 
problems 


190 


Types of weaknesses in a series 
of practice problems 


191-195 


Revisions to make in a series 
of practice problems 


196-201 




















J. 2. 5 
IDEMTIFICATION 
MATRIX 



SOURCES OF INFORMATION ABOUT WEAKNESSES 
OF A SEQUENTIAL SERIES OF PRACTICE PROBLEMS 



SOURCES 



CRITERIA 



ERIC 



ERROR data 
on PROGRAM 



From: 



Ul) 



*An identification of 
a series of program 
problems on which a 
relatively high 
frequency of errors 
has ocaiAVi-ed 

*An identification of 
a series of crvors 
in which: 

••Errors occur only 
on a. number of 
sub- criterion 
problems but HOT on 
the preparatory 
progression 
problems leading up 
to then 

OR 

• •Errors occxw* on the 
preparatory 
progression 
problems and the 
Quh'criterion or 
criterion problems 

-An identification 
(independent of error 
data) of what a 
series of program 
problems was designed 
to teach 

• •Specific component 
skills 

• •Combination of 

sJ^.i I Is 



ERROR data 
on TESTS 



From: F0B4 J. 2(2) 



^ GROUP error data on 
items traceable back 
to a series of 
progran} problems 

• •Error data on a 
nmber of test 
it 0ns 

• •RECALL/TRANSFER 
error data 



From: F0RI4 J.2(3) 



-- INDIVIDUAL error 
patterns traceable 
hack to a series of 
program problems: 

"Diffic74lties with 
specific component 
skills identified 



INTERVIEWS 
and /or 
OBSERVATIONAL DATA 



Interviews or 
observation provide 
infomation about 
difficulties students 
have with a series of 
program problems 
(See page 184) 



190 



J.2.t> 

DECISION 
MATRIX 



DATA 
AVAILABLE 



ACT I ON 
TO TAKE 



EXAMPLES 



IDENTIFYING THE NATURE OF PROGRAM WEAKNESS 
BASED ONLY ON ERROR DATA ON PROGRAM PROBLEMS OR TASKS* 



H?gh error rate both on: 
••Preparatory problems 
and on 

••Sub-criterion problems 
(which fol low them) 



INTERPRET 

"Multiple emissions J 
or 

"Multiple and vaxn^ed 
wrong responses 

as evidence of: 

-An exoessivelij lean 
progression: 

•'Insufficient cuing 
of responses to 
examples provided 

"Insufficient number 
of prograsn problems 
(offering different 
examp les ) 

"A less than optimn 
sequence of problem 
examples 



error rate 



•Only on a series of 
sub-criterion prob- 
lems (or criterion 
problems) 

•But NOT on the 
preparatory 
p rog res s i on p rob 1 ems 
which precede them 



INTERPRET 

"i'fultiple emissions^ 
or 

"tfultiple and varied 
wrong responses^ or 

• 'i'fultiplej systematic 
wrong responses 

as evidence cf one or 
more of the followifig: 

-Too rapid fading of 
ones before 
suh-criterion problems 

"Failure to provide the 
typos of or number of 
problems relevant to 
sub'Criterion (or 
criterion) beliaoior 

-Failure to provide 
practice with TRANSFER 
problems that could 
facilitate TRANSFER 
to sub-criterion 
problems 

-A program sequence 
which is excessively 
NON- lea>\ (i,e. ^ too 
nuch assist^ince is 
provided) 



High error rate on: 



•A series of program 
problems which do 
not end in a 
sub-cr 1 terion 
behavior (tiiere are 
a':idi tional problems 
before sub-cr I terion 
problems are reached) 



INTERPRET 

"tfultiple omissions J 
or 

"tfultiple and varied 
wrong responses^ or 

"^^tultiplcj systematic 
wi*ong responses 

as evidence of one or 
more of the folloioing : 

-An excessively lean 
sequence: 

'Ins-T4fficient cuing 
to problems 
provided 

-A less than optimum 
sequence of vroblems: 

• '^>eauence of 
exmples is less 
then optimum 



e.g., 

-The. onouwt 04 ^t/itngtli 

Oi5 CtlinQ iO'l ^ic 

bidlMi^aal p^obleiu 
in -C/te 04 tiio. 

fading ctUng i6 
made too ahnupttij 

-The. nuinbo/L of^ pfLoblem6 
provided 'Ci too iimcUZ 
to enable. tlnL ^tudmit 
to be ohlc to pKOQKC^^ 
to tlid 6idj'CAAXtnA.on 
iviob im 



-T/ie vo^ixc-tf/ oi 
ejcmvlQA> LL&Q,d iii the. 
practice. p^oQKeAi^ion 
vli too 6mail oh. tivLn 
to unabZe. t!iz ^tiidciit 
to TRAWSFER 6hUm to 
nei') (LKajnpltii pfiovidtd 
in 6ab-cAiteAion 
ptobizm 

-Too mack cut^ihtaiicc. 
ii> pfwvidzd and 
Haded oiit cit cdiZ. 
[actoanting f^oH. tov) 
e/L/L0'^4 on pKepoAcitoK.ti 
pKobtesnii and \iig\i 
C/IA044 o\i -6ub- 
cAiXeAlon piobte.mii 



e.g., 

occLiA in a sequence. 
be.<ion.e ea&ieA one.^ 
(-t.e., exanp^.e/:. Lvhidi 
^ueqiuAe mfie dii^^itatt 
tAan&^en. com ejxkzie' 
■tlian one^ lOkicJi. 
xcquOie t(>^6 dUf^iattt 



*fdent I f I cat i on of program weaknesses can be even more dependably made wht:n 
test date (See next pagp) confirm the interpretations rrtade on the basis of 
program data. 



191 



DECISION 
MATRIX 



IDENTIFYING THE NATURE OF WEAKNESS IN A SERIES 
OF PROGRAM PROBLEMS BASED ONLY ON "GROUP'' TEST DATA 
(RELEVANT TO PROGRAM PROBLEMS ON WHICH ERRORS OCCURRED) 



TEST RESULTS 



ACTION 
TO TAKE 



GROUP error scores 
on tests: 

•Frequent omissions 

or 

•Frequent var iable 
wrong answers 



INTERPRET 
as evidence of: 

■'An exci Hvelij lean 
progression due to 
one or move of the 
foltoDing: 

• • Insuf fidently 
strong cuing 

• *An insufficient 
number of examples 
or problems 

'•Premature or abrupt 
fading of cues 



GROUP error scores 
on tests: 

•Frequent systematic 
wrong answers 



INTEFPRET 
as evidence of: 

-An excessively lean 
progression due to 
one or more cf the 
follouing: 

• 'Insufficient 
variation in 
examples or 
problems 

^Provision of 
mis leading cues in 
program problems: 

••Use of atypical 
examples 

Ambiguous statement 
of rulef^ pertaining 
to problems 



GROUP error scores 
on tests: 

•Few errors on 
RECALL items 

•Frequent errors on 
TRANSFER items 



INTERPRET 
as evidence of: 

•-An excessively lean 
progression due to 
one or more of the 
fencing: 

• • Insufficient 
variation in 
examples or 
problems provided 
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J. 2. 5 

DECISION 
MATRIX 



IDENTIFYING THE NATURE OF WEAKNESS IN A SERIES OF PROGRAM PROBLEMS 
BASED ONLY ON »»l ND I V I DUAL»» TEiiT DATA REVEALING DIAGNOSTIC PATTERNS 
(RELEVANT TO PROGRAM PROBLEMS ON WHICH ERRORS OCCURRED) 



DIAGNOSTIC 
PATTERNS 



Diff ?cul ties 
indicated for 

DISCRIMINATIONS 



Difficulties 
indicated for • 

GENERALIZATIONS 



Difficulties 
indicated for 

ASSOCIATIONS 



ACTION 
TO TAKE 



INTERPRET 
as evidence oft 

-Program failure to do 
one or more of the 
following: 

^•Contrast INPUT 
classes 

••Identify the basis 
for the difference 
between classes 

••Identify the 
boundaries between 
(limits) classes 

••Provide sufficient 
cuing to enable 
discriminations 
to be made 

••Provide enough 
examples of 
differing classes 



INTERPRET 
as evidence of: 

-Program failure to do 
one or more of the 
folloioing: 

••ShoD similarities 
between members of 
each INPUT class 

• 'Identify the basis 
for the similarity 
between members of 
each class 

••Identify the limits 
of each class 

• •Provide sufficient 
oiling to enable 
generalizations 
to be made 

• • Provide enough or 
varuing examples 
within each class 



INTERPRET 
as evidence of: 

-Program failure to do 
one or more of the 
following: 

• •Identify the basis 
for associating 
INPUT class and a 
given ACTION 

••Px'ovide sufficient 
cuing to enable 
associcctiiyns to be 
formed 

••Provide sufficient 
practice problems 
or examples to 
enable associations 
to be fomed 
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J. 2. 5 

DECISION 
MATRIX 



IDENTIFYING THE NATURE OF WEAKNESS IN A SERIES OF PROGRAM PROBLEMS 
BASED ONLY ON "DIAGNOSTIC" TEST DATA AND ON INTERVIEW/OBSERVATION 
(RELEVANT TO PROGRAM PROBLEMS ON WHICH ERRORS OCCURRED) 



DATA 



DIAGNOSTIC tests identify 

difficulties wi th 
specific component skills: 

• •Discriminations 

• •General i zations 

• •Associations 



ACTION 
TO TAKE 



INTERPRET as evide^e of: 

-Program failure to deal with 
these skills 

(See page 193 for possible program 
failures which can account for 
each type of difficultij) 



INTERVIEW/OBSERVATION 
results: 

••Student questions 

••Student appearance of confusion 

••Student doing something other 
than what instructed to do 



INTERPRET as eviiienoe of: 

-Program failure to do one or more 
of the following: 

••Gear language difficulty 
to target at/dience 

••Provide for adequate control 
of attention^ observation 

••Provide unambiguous instructions 
about tasks to be verformed 
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J. 2. 5 

IDENTIFICATION 
MATRIX 



THREE MAJOR TYPES OF PROGRAM WEAKNESS 
WHICH HAVE TO BE ELIMINATED* 



TYPES OF 
PROGRAM WEAKNESS 



CRITERIA 



(1) 

LEANNESS 
of program 



' Excessively_ lean: 

•'Insufficient cuing 
in individual 
problems 

••Too abrupt fading 
of cues in series 
of problems 

••Insufficient number 
of program problems 
or examples 

- Insufficiently lean: 

••Cuing maintained 
too long (not faded 
soon enough in 
series) 

••No uncued problems 
provided 



(2) 

CONFUSIONS 
created by program 



'Vocabulary J sentence 
structure J etc.^ 
inappropriate for the 
target audience 

'Omission of or 
ambiguous statement 
of: 

• •Objectives 

--Task instructions 

- -Directions to 
control attention 
or observation 

-Examples used: 

• -Atypical 

- -Sequence of 
examples (difficuti/ 
easy rather than 
other Way around) 

■-Cues used: 

--Call attention to 
wrong INPUT or 
ACTION features 



(3) 

SPECIFIC SKILLS 
not accommodated 



-Techniques 
appropriate to 
teaching of specific 
skills are: 

- -Omitted 

• •Incompletely 
implemented 



*A1 1 these types of weakness may coexist in varying combinations. Program 
revision will, therefore, have to deal with any or all of them which 
characterize a series of program problems. 



SEE PAGES 


196-197 


198-199 


200-201 
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J. 2.5 

DECISION 
MATRIX 



(1) DETERMINING HOW TO REVISE A SERIES OF 
PROGRAM PROBLEMS TO OVERCOME INAPPROPRIATE DEGREE OF LEANNESS 



PROGRAM 
WEAKNESS 



ACTION 
TO TAKE 



EXCESSIVE 
LEANNESS 



'Deal with the series of problems 
before looking to individual 
problems: 

• •Reduce the abruptness of 
fading: 

/Maintain some type of cuing 
for more problems than before 

/Increase the strength of 
cuing (where cuing is already 
provided) 

••Increase the number of problems 

'Deal with individual problems 
making cuing strength appropriate 
to each problem and where it is 
in the series 

(See pages 186-187) 



INSUFFICIENT 
LEANNESS 



-Add some program problems which 

••Have less cuing strength than 
the ones already in the 
program 

••Provide an opportunity to 
fade out cues gradually^ and 

••Provide an opportunity for 
student practice of problems 
like the sub -criterion (or 
criterion) problems; i.e.^ with 
no cuing available 



EXAMPLES 



ERIC 



SEE OPPOSITE PAGE 



196 



J. 2. 5 

EXAMPLES 



EXAMPLES ILLUSTRATING REVISIONS 
WHICH OVERCOME PROGRAM WEAKNESS DUE TO "LEANNESS"* 



#1 

EXCESSIVE 
LEANNESS 




ORIGINAL 



b. 



0«tcrlb« th« direction tn 
which th« four forces %t% 
being appUad to the box 
■nd alto the tide of the 
box 00 Mhlch each one It 
being applied. 

A bcotq apt)tUd tht 



When a tign Instructs 
train passengers to pull 
the emergency cord» whet 
Is It Instructing pessengers 
to do7 

To apply A datinfiOAd ioKtt 
to U 



Uit oi tht box to tkt flight 



C U bting aopUtd Utom tkt 



^kt in a It it-Hand 



dOutttion 



P iA beom egppUtd jKam 
bttM in a daawKJui 



C. 

Whet do we meen by the 
direction of e force? 
Give two contrestlng exafl»ples 
different from the ones used 
here. 

fHhtthtA a ptuh o\ a putt 
opptUd to tkt ohjtct l& 
touKUid tht Aight, UU» 
upf OK doutn 



REVISION involves: 

The a ddition of the following 

p rob 1 ems : p sug,; 

"Before (a) : 




In Nhlcit rilractlon l« 
hm ■ •pptyinf a f«rnT 



In a 4^>U'fcajni diiuUiM 

FraN .«Mdt tltta ot tha 
H«fon vra Nan A a«W Ha«i I 
•Wirlni th« fsrcaT 

A tht Uttf B i^m 

What are some of the directions 
In which a force can be applied 
to an object? 

Up, dqm, KAj^kt, Iz^t 

When the sergeant tells recruits 
to carry their own bags, what 
kind of force is he telling them 
to apply to the bags? 

A t^o^ce apptizd In an VLp^a/id 



#2 . 

INSUFFICIENT 
LEANNESS 



ORIGINAL 



□ 

1 



□ 



f 

□ 



write In the letter in the 
epproprlete bUnk Indlcetlng 
the dTrectlon of the force. 



C to the right 
B u pwerd 



p to the left 
^ downwerd 



b. 



REVISION involves: 

The addi tion of the following 
problems: 

- Fol lowing (c) : 

What are some of the directions 
in which a force can be applied 
to an object? 

Up, dom, Aytgkt, l^{^t 

What do we mean by the direction 
of a force? Give two contrasting 
examples different from the ones 
used here. 

W(te;tKeA a pa&fi a pu££ apptizd 
to thz object ^ tMo/id the 
Ayigkt, It^t, up, OH. dom 




The two forces 
shown ere 
ecting in 

which two 
directions? 




The two forces 
shown ere 
ecting in 

which two 
directions? 



A Hlght'hami diAtction and In an upDoAd and damooAd 
tt^t^hand diAtction dixtctlon 



*Due to space limitations and for the purposes of illustrating a pointy the 
examples provided below are artificially exaggerated; in ffl ^'excessive'' is 
overexaggerated^ and in #2 '^insufficient^' is over e xaggerated. 
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J,2-5 

DECISION 
MATRIX 



(2) DETERMINING HOW TO REVISE A SERIES OF 
PROGRAM PROBLEMS TO OVERCOME CONFUSIONS CREATED 
BY THE SERIES OF PROBLEMS 
(On this and on opposite page) 



PROGRAM 
WEAKNESS 



ACTION 
TO TAKE 



Program poorly planned 
for target audience regarding: 

••Vocabulary (low frequency words) 

• 'Sentence length, structure 



-Reduce difficulty level of 
vocabulary: 

••Replace lonr j wovdc with 
short words 

* •Replace low frequency words 
with high frequencij words 
(consult word frequency tables) 

-Reduce difficulty level of 
sentences: 

••Reduce the length of sentences 

••Simplify the grammatical 
structure of sentences 

••Keep the number of sentences 
low 

••Eliminate words or devices 
which are irrelevant to the 
task at hand 



Omi ss ion of 
or ambiguous statement of: 

' • Object i ves 

•Task Instructions 

•Directions to control attention 
or observation 



'-Take care of *^omissions" by 
providing necessary statements 
or techniques 

Sirtplifyj clarify j or strengthen 
all statements or techniques 
designed to guide: 

••Task performance 

• •Attention and observation 
behavior 



EXAMPl ES 




In Mhtch dlr*clion It 

frtm ■«(lc^ ltd* of th« 
*»*fv(ng tfM forcmi 



e,g. 



ORIGliNlAL: 



REVISION: 



[lUjuJ^iKotzd] a/iz bting applied to tliz box and oJtMo thz 
6ldz Oj$ thz box on loklak exich one. AJi being apptizd. 



apftced to tivU box 
[UZmtAotzd) . In 
iolitch dAJizcjtion am tach 
j^o/Lce being applLzd? 
To iA)lu,cii 6ldz ofi tilt 
box l6 zach i5on.ce being 
apptLtd? 



REVISION: Add to the original: 

Hotz tkt di/izctijonA in 
voliicii each oaam iM 
pointing • 

kJlMo notz on lokidi 6idz 
oj5 tSie, box tack a/Uia^ 
Lb placed. 



ERIC 
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J. 2. 5 

DECISION 
MATRIX 



(2) DETERMINING HOW TO REVISE A SERIES OF 
PROGRAM PROBLEMS TO OVERCOME CONFUSIONS CREATED 
BY THE SERIES OF PROBLEMS 
(On this and on opposite page) 





Examples used: 


Cues are: 


PROGRAM 
WEAKNESS 


••Atypical or low frequency 

••In an Ineffective sequence 
or order 


••Misleading (calling attention 
to wrong INPUTS or wrong 
properties of INPUTS) 

••Ambiguous, unclear 


ACTION 
TO TAKE 


'-Replace examples less capable 
of eliciting responses with 
those more capable of doing so 

OR 


^Simplify, clarify^ or use neio 
cues to: 

••Call attention to relevant 
properties of INPUT or ACTIONS 




^Reorder the sequence of 
appearance of those examples: 

••Put concrete examples earlier 
in sequence before abstract 
ones 


••Call attention to what are not 
relevant properties of INPUTS 
or ACTIONS 




••Put visual examples before 
verbal examples 





EXAMPLES 

(See example 
on page I98) 



e.g., 

ORIGINAL: Ve^cAihz tkz cUjizction In i^hlch tkz f^ouA f^oKceA 

[Itlu^tA^td] oKd be^np apptizd to HlQ. box and ai&o tlit 
6ldt oi iht box on whXch zach owe X6 bzing applLzd. 

REVISION:^'' REVISION: Add to original 

Note, that an oAAOi^J 
Kzpfie/iZYvtinq a paZting 
j^o/tee and om /tep/ie^eKLt- 
Inq a pti&kinQ ^okcz oaz 
ix^aalZvf apptiod to 
dZf^jeAent 6ide^ o{^ a 

In Mhich dtraction is bOX* 
Han 8 applying a fore*? 

Jn a fUght-harui diAtction 

in which direction is 
Han A applying a forca? 

In a iuQht'hand diAtction 
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DECISION 
MATRIX 



PROGRAM 
WEAKNESS 



ACTION 
TO TAKE 



(3) DETERMINING HOW TO REw!SE A SERIES OF PROGRAM PROBLEMS 
TO OVERCOME THE FAILURE TO PROVIDE APPROPRIATE PRACTICE NEEDED 
FOR THE ACQUISITION OF SPECIFIC SKILLS 
(On this and on opposite pages) 



DISCRIMINATIONS 
••Incompletely treated 
••Inadequately treated 



-Do one or more of the following : 

--Contrast INPUT classes 

Identify the basis for the 
difference between classes 

-'Identify the boundaries 
(limits) between classes 

--Provide sufficient cuing to 
enable discriminations to be 
made 

--Provide enough examples of 
differing] classes 



GENERALIZATIONS 
Incompletely treated 
Inadequately treated 



'Do one or more of the following: 

--Shao similarities between 
members of each INPUT class 

--Identify the basis for the 
similarity between members of 
each class 

- - Identify the titrrits of each 

class 

--Provide sufficient cuing to 
enable generalizations to be 
made 

- -Provide enough or varying 

examples within each class 



Mi 





Class ! 




Class 


II 




Class III 




EXAMPLES 






























fae;&oeen tlie. c^a64a6 of^ Izvvu^ 
coKiA^-tA of^ thz position KoZatioz 
to one. ojioihoA of^: (I) </ie Zoad; 
[2] tiiz zUo^t ioA.cz; and 13] thu 

OR 

"A dia^inivn contOA^wing aJUi tiiKta 
cJbi66z^ (06 above.) couZd be pn.o- 
vidzd to g-cve a cont/ia^tCng ovzk- 
vlei^ of. the thA.e,e, cttiji^c/ieiot 


"In M C£a64 II ZeveM, tiiz load 
Aji bQjtx\)zzn the. Q.f^(^ofit (^okcc and tke. 
f^iilcAm. A tvhe.eZbaAJioio l& anotlieA 
e.xampZe. o{^ a Cla^^ II IcveA.'' 
OR 

A dZagKom 6hwZnQ maiJxptz e.xmplQA 
can provide an oveAvieiO of^ a 
^Znglc cta^6* 
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DECISION 
MATRIX 



(3) DETERMINING HOW TO REVISE A SERIES OF PROGRAM PROBLEMS 
TO OVERCOME THE FAILURE TO PROVIDE APPROPRIATE PRACTICE NEEDED 
FOR THE ACQUISITION OF SPECIFIC SKILLS 
(On this and on opposite pages) 





ASSOCIATIONS 


CHAINS* 




PROGRAM 
WEAKNESS 


••Incompletely treated 


• • 1 ncompletely 


treated 


••Inadequately treated 


• • 1 nadequatel y 


treated 



ACTION 
TO TAKE 



'Do one or more of the following: 

••Identify the hasis for 
aasoaiating INPUT class and a 
given ACTION 

••Provide sufficient cuing to 
enable associations to be 
formed 

* •Provide sufficient practice 
problems or examples to enable 
associations to be formed 



^Do one or more of the following: 

• •Provide sufficient number of 
opportunities to practice the 
whole chain 

••Provide cuing to assist the 
learner to remember the 
sequence in which parts of the 
chain are to be performed 



*Failure of students to produce a chain is not inferred; rather, it is directly 
observed by an observer as the student exhibits procedures or there is an absence 
of or an error in the "work" (on paper and pencil). 



EXAMPLES 



Since the designation of a lever 
class (i.e., I, II, Ml) is 
arbitrary (i.e., a matter of 
convention) and not based on 
conceptual properties, additional 
practice in associating labels and 
examples should be provided. 



In chaining verbal statements about 
definitions of lever classes (in 
addition to practice involving 
discriminations , general Izations , 
and chains) some practice in pro- 
ducing the whole chain should be 
provided (e.g., stating spatial 
relationships between load, 
fulcrum, and effort force). 
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REVISING THE TOTAL INSTRUCTIONAL PROGRAM 





page 


Information sources to review and 
types of weaknesses which can be 
identified for a total series of 
program problems 


20^ 


Sources of information about 
weaknesses of a total program 
(delayed testing) 


205 


Identifying the nature of program 
weaknesses based on results of 
delayed testing 


206 


Revising the total program to 
overcome weaknesses identified by 
means of delayed testing 


207 











* 



The preceding section on revising a 
sequential series of program problems 
Is applicable to the "revision of the 
total program." Accordingly, it will 
not be repeated. 

To be added to the guidelines presented 
In the preceding section are some 
considerations about program weaknesses 
identified (1) by delayed testing, and 
(2) by the administration of program 
components in alternative sequences. 
The first of these follows on the next 
several pages. The second follows in 
the next section. 
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SOURCES OF INFORMATION ABOUT WEAKNESSES OF A TOTAL PROGRAM 





Difference in ERROR data on tests 


Difference in ERROR data on tests 




administered 


administered 


SOURCES 


BEFORE and IMMEDIATELY AFTER 


IMMEDIATELY AFTER 




instruct ional program 


the instructional program and on 






a DELAYED BASIS* 



J. 2. 5 
IDENTIFICATION 
MATRIX 



CRITERIA 



Frorn: FORM J. 2(2) 



' GROUP ervov data on all test iterno related to 
each suh-ariterion behavior and alj^ test items 
related to the ariterion behavior 



Fron: FORId J. 2( 3) 

^INDIVIDUAL error patterns on all test items 
related to each sub-ariterion behavior and 
all items related to the ariterion behavior 



••The difference in error data 
obtained before and irmediately 
after program administration 
serves to identify program 
weaknesses in allowing student 
aoQuisiticn of: 

/Siib'-ariterion and criterion 
behaviors 

/Component skills 



••The difference in error data 
obtained, immediately after and 
on a delayed ha:iis after program 
administration serves to 
identify program weaknesses in 
allowing student retention of: 

/Sub^criterion and criterion 
behaviors 

/Component skills 



*Delayed testing should be performed during the last tryout cycle when thr* 
instructional program is more likely to be capable of producing acceptable 
"acquisition" levels. 
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J,2,5 
DECISION 
MATRIX 



IDENTIFYING THE NATURE OF PROGRAM WEAKNESS 
BASED ON RESULTS OF DELAYED TESTING 



DATA 



ACTION 
TO TAKE 



On TOTAL test for: 

••Sub-criterion behavior 
or 

••Criterion behavior 

-IMMEDIATE results show low error 
rates 

-DELAYED results show relatively 
high error rates 



INTERPRET as evidence of: 

'Program failure to provide 
sufficient practice of the 
sub-criterion or of the criterion 
behavior 

'The omission may involve one or 
more of the fotlouing: 

••Absence of sufficient 
repetition and/or review with 
old examples 

* •Absence of repetition and/or 
review with new examples 



On test items related to: 

••RECALL/TRANSFER 
or 

• •Speci f ic ski 1 Is 

-IMMEDIATE results show low error 
rates 

■DELAYED results show relatively 
high error rates 



INTERPRET as evidence of: 

-Program failure to provide 
sufficient practice of the 
component skills 

-The omission may involve one or 
more of the following: 

• * Absence of sufficient 
repetition and/ or reviei') with 
old examples 

••Absence of repetition and/or 
review with new examples 
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J. 2. 5 
DECISION 
MATRIX 



DETERMIWIKG HOW TO REVISE A TOTAL PROGRAM 
EXCESSIVELY LEAN FOR ''RETENTION'' PURPOSES 



PROGRAM 
WEAKNESS 



ACTION 
TO TAKE 



Insufficient amount 
of practice of: 

••Sub-criterion behavior 

••Criterion behavior 



Do one or more of the following: 

••Add more practice probleme at 
the end of a progression 
relating to a suh^criterion 
(or to a criterion) behavior; 
include: 

/Old examples 

and/or 
/iVeu) examples 

• •FolloDinrj practice with other 
sub'-criteTion behaviorsj 
introduce review problems 
which can include: 

/Old examples 

and/or 
/New examples 



Insufficient amount 
of pr "ictlce of: 

• •Component ski \ Is 

••TRANSFER problems 



Do one or more of the follouing: 

••Intersperse more practice 
p-i*oblems relating to the skills 
not retained in the middle of 
the progression which is 
building up to a sub^cviterian 
or criterion behaoior: 

/Preferably new examples 

••Insert the problems in that part 
of the series of problems 
devoted to a particular skill 

••If two or more component skills 
were taught separately^ consider 
inserting items dealing with one 
skill at a time (particularly if 
retention failure is serious) 
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REVISING THE SEQUENCE OF PRACTICE* 





page 


Two types of sequencing issues 


210 


Sources of Informatfon about 
sequencing weaknesses 


211 


Revising the sequence of 
practice 


212 















Review pages ISS-ISS. 




J. 2. 5 

IDENTIFICATION 
MATRIX 



TWO TYPES OF SEQUENCING ISSUES 



SEQUENCING 
ISSUES 



Within a 
S INGLE program 
devoted to one criterion behavior 



CRITERIA 



'Haoe the separate 8ub~a2*iterion 
behaviors, all of which relate 
to a single criterion behavior, 
been taught in the right 
sequence; i*e*, has tlte sequence 
been so arranged that the 
learning of one sub-criterion 
behavior facilitated the learning 
of others; and, in turn, has the 
learning of the criterion 
behavior been facilitated by the 
order in which its constituent 
sub-criterion hehaoiors haoe 
been learned 



Between TWO or MORE programs 
devoted to two or more 
cri ter ion behaviors 



'Have programs devoted to two or 
more criterion behaviors been 
sequenced so that the learning of 
one has facilitated the learning 
of one or more of the others 



Tde uiitwion bcAau^o/L InwHsjoM 
Hit ^elcutio^i^lvLp be^oeen batanco^d 
EXAMPLES and unbalanced {^OKced and mov^ridnt 

0^ obj^cX^ 

StvtKol ^iib-cJuZoAion bdxavlou 
hav^ bam -tdeiti^f^^ed: 

Vi/iecfyLon OjJ i^o/lcc 

ObjtcU In movmnYit 

Obj^(iti> aX /i^t 

Balanced/ unbaJtancQ^d f^oKCQ, 

-The issue is: Is the order in 
which each of these subrcr i ter i on 
behaviors has been learned the 
optimum one for facilitating the 
learning of them all? 



boJiavA^o^^ to be, taught (O/vcc/i 
involve, t/ic6c concepts • 



• FO/LCC 



• •Mac/u.itC/6 



-The issue is: Is the order in 
which each of these cri terion 
behaviors has been learned the 
optimum for facilitating the 
learning of them all? 
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J. 2. 5 

DECISION 
MATRIX 



WHEN TWO DIFFERENT SOURCES ARE LIKELY TO BE USED 



SOURCES 
OF INFORMATION 



EXPERIMENTAL 



STATISTICAL 



ACTION 
TO TAKE 



'Consider using the experimental 
approach in assessing sequencing 
adequacy only_ during INFORMAL 
try out: 

• •STEP J. 1 



'Consider using the statistical 
approach both during INFORMAL 
and DEVELOPmNTAL try out: 

• 'STEP J. 1 



and 



•STEP J. 2 



RATIONALE 



er|c 



-The experimental approach to 
assessing sequencing adequacy 
can provide more dependable 
results on which to base 
sequencing decisions 

-It is, However, time-consuming 
and does dilute the size of the 
sample 

-However, when time and student 
availability permit. It should be 
considered in both informal and 
devel opmental t ryouts 



211 



J. 2. 5 
DECISION 
MATRIX 



DETERMINING HOW TO REVISE PROGRAM 
IN THE LIGHT OF SEQUENCING DEFICIENCIES* 



RESULTS 



Experimental resul ts Identi fy : 

-The administered sequence 
producing: 

••Lowest average time to complete 
separate programs (or program 
section) 

••Lowest average error rate on 
programs 

••Lowest average error rate on 
tests 



Statistical results Identify: 

-Groups scoring high and low on 
one program: 

••Time to complete two or more 
programs 

••Errors on programs 

• •Errors on tests 

-Comparison with these yroups on 
another program reveal whether 
there is a facilitating effect 



ACTION 
TO TAKE 



-Alter the sequence of praatiae in keeping with 
the sequence favored by reaiilta 



^Applicable to sequencing of sub-criterion behaviors related to a single 
criterion behavior or to sequencing of separate criterion behaviors. 
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JOB PROCEDURES 





page 


Information sources to review 
and information to look for 




Priorities to observe in making 
program revisions 


215 


SUMMARY OF PROCEDURES 


216 


Adequacy of the revision process 


217 
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J,2.5 
IDENTIFICATION 
MATRIX 



INFORMATION SOURCES TO REVIEW 
AND INFORMATION TO BE OBTAINED 



SOURCES 



CRITERIA 



FORM J,2(2) 

Errors 
on programs 



-Error rate for 
program uhick 
iAipMtifies: 

••Extent of revision 
needed: 

/Total program 

/Series of program 
problems 

/Isolated program 
problems 

••Specific types of 
revisions to make: 

/To overcome 
excessive leanness 

/To overcome 
insufficient 
leanness 

/To overcome 
confusions 

••Sequencing failures 



FORM J-2{0 

Errors on tests 
(for GROUPS) 

FORM J ,2 (3) 

Patterns of errors 
for INDIVIDUALS 



-Error rate on tests 
identifies: 

• •Acquisition an/l 
retention 
difficixlties 

/Sub -criterion, .^.nd 
criterion hahx^ior 

/Component skills 

••T]ipes of errorr: 

/OnissionC: 

/Systematic iwoig 
answers 

/Varidhlc wrong 
answers 



DIAGNOSTIC 
test results 

Resul ts of: 

• • I ntervi ev;s 

• •Observation 



-Error rates on tests 
or results of 
interviews or of 
observation identify: 

• • Spc cifi c ski lis 
not learned 

• -Penturrs of program 
hi^iich contribute 
to d,ifficiitt]j : 

/Voca})ulm*y y 
sentence structure 

/Instructions about 
tanks or vhat to 
observe 



ERIC 





PRIORITIES 


TO 


OBSERVE 


WHEN LARGE SCALE REVISION 


J. 2. 5 


IS TO 


BE 


MADE (1 


.E., IN TOTAL PROGRAf^) 


DECISION 










MATRIX 











PRIORITIES 


1st priority 
occurring In the 
FIRST 
revision cycle 


2nd priority 
occurring in the 
SECOND 
revision cycle 


3rd priority 
occurring in the 
THIRD 
revision cycle 


ACTION 
TO TAKE 


'Keep ''the AhfOVllT of 
criterion behavior 
practiced at the 
same time'' unchanged 

'Consider one or more 
of the following 
approaches : 

••Strengthen tJte 
degree of assistance 
to the program 
problems already in 
une secj[iience emu/ or 
remove assistance 
less early in the 
sequence 
OR 

• 'Introduce new 
program problems to 
the sequence provid- 
ing more examples 
(new or repetitions) 


IF THE SECOND 
TRYOUT STILL PRODUCES 
ERRORS ON A SERIES 
OF PROGRAM PROBLEMS: 

'Keep "the A-JOUNT of 
criterion behavior 
practiced at the 
same time" unchanged 

'Definitely add ne^o 
program problems 
appropriate to the 

students have 
difficulties 
(Intersperse the neto 
problems among the 
existing problems) 

'Also consider 
strengthening the 
degree of assistance 
for existing problems 


IF THE THIRD 
TRYOUT STILL PRODUCES 
ERRORS ON A SERIES 
OF PROGRAM PROBLEMS: 

'Reduce the amount of 
criterion behavior 
practiced at the same 
time (one level lower 
on each successive 
revision) 

'Also consider adding 
YLPh) ryY*nnram tyr^oblems 



EXAMPLES 



SEE SECTION "I," PAGES 73-96, 
FOR EXAMPLES OF PROGRAM SEQUENCES 
DIFFERING IN "AMOUNT" OF CRITERION BEHAVIOR 
PRACTICED AT THE SAME TIME 



215 




J. 2. 5 



ILLUSTRATION SUMMARIZING PROCEDURES 
INVOLVED IN REVISING INSTRUCTIONAL PROGRAM 



#1 



n 



REVIEW 

a. Error data for 
program: FORM 
J. 2(2) 

b. GROUP error data 
on tests; 

FORM J. 2(1) 

c. INDIVIDUAL error 
patterns on tests: 
FORM J. 2(3) 

d. Diagnostic test 
results 

e. Interview and 
observational 
resul ts 



IDENTIFY 

a. Extent of program 
weakness 

b. Nature of program 
weakness regarding: 

• -Sub-cri terion 

and criterion 
behaviors 

• 'Component ski 1 Is 

••TRANSFER 
requi rements 

• •Retention 
requi rements 

• •SequencI ng 



REVISE 

Program to 
overcome 

identified weakness 

Program observing 

priorities for 

keeping program as 

lean as possible 

consistent wi th 

acceptable 

ach i evement 1 evel s 



FORM J.iid) 



FORM J. 2(2) 



FORM J. 2(3) 



+ -H 



'm\m\m\m *5?~ "ij 
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J. 2. 5 

STANDARDS 
MATRIX 



ADEQUACY OF PROGRAM REVISION 



PROPERTIES 



CRITERIA 



DATA- BASED 



-To the extent 
possible, spedfio 
program wedknesses 
are identified by 
direct evidence and 
by inference from 
data: 

••Student performance 
on the instmctional 
program itself 

••Student performance 
on criterion tests 
and on diagnostic 
tests 

• •Results of 
interviews and 
observation 



COMPLETENESS 



--Revisions in program 
are addressed to 
failures in all three 

• * Acquisition 

••Retention 

••Transfer (to new 

examples) 

^Weaknesses removed 
involve: 

••Content (amount and 
type of practice, 
etc* ) 

• •Format ( vocabulary, 
etc. ) 

••Behavior control 
features 
(Instructions, 
etc. ) 



MAGNITUDE 



-Revisions made adhere 
to priorities in 
amount of change 
required: 

••Individual program 
problems 

• •Series of problems 

••Total program 

(No more change is 
made than is 
required) 

'-Revisions make no 
changes in the amount 
practiced (at the 
same time'' except 
when later stages of 
tryout reveal them 
to be necessary 
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STEP I J. 2 



COMPLETION CHECKLIST 





IDENTIFIED 


PERFORMED 


PRODUCED 


FORMS COMPLETED 




Entering proficiency 


Administration of a 




FORM J. 2(1) 


J. 2.1 


of developmental 


"before" test 






tryout sample 











Errors on program 


Administration of 




FORM J. 2(1) 


J. 2. 2 




Instructional 








program 







J. 2. 3 



Errors on tests 


Administration of 




FORM J.2(l) 




"after" tests 




FORM J. 2(2) 












FORM J. 2 (3) 



J. 2. 4 


Types of learning 
failures 


Diagnostic tests 
and interviews 
















J. 2. 5 


Program weakness 




Revised program 
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STEP I J. 3 



J. 3 



Conduct a field test of the Instructional program. 



, - , .Administer pre- and post-tests and the instructional program to a 
J.J. I 

large sample of the target population. 



J. 3. 2 



Report results of the field test. 
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STEP i J. 3 i 



OVERVIEW 





INPUT 




ACTtON 




OUTPUT 




Tests 
and 

cyclically revised 
and acceptable 
program 


► 


Administer to 
large sample of 
target population 




Reliable evidence of 
the effectiveness 
of the instructional 












program 





J. 3.1 



Tests 
and 
revi sed 
inatructioneil program 



J, 3.2 



Test results 



iv 



-Admin is t-<?r «^nd score 
"before and after tests 
AND 

-Administer program to 
a large sample of 



target population 



ii 



Proficiency levels 
for sample 



iii 



Computo tud record 
proficiency levels 
attained 



Identification of 
percentage of students 
attaining predetermined 
standards of proficiency 



vi 



STEP 



J. 3 





PAGE 


INDEX 




CRITERIA FOR 
IDENTIFYING INPUTS 


ACTION.TO BE TAKEN 


STANDARD FOR OUTPUTS 


FORMS TO USE 






-MATRIX: Adequacy 
of field test 
administration .229 


FORM J. 2(2) 
SUMMARY OF 

PROCEDURES ... 228 







-MATRIX: Adequacy 
of repurt of 
results .'. . .235 


SUMMARY OF 
PROCEDURES ... 23^ 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


An identification of the proficiency level attained 
by a Large sample of the target audience (n of 
approximately l00'-200 students) , 


WHAT YOU WILL 
WORK FROM 


l) Sample of target audience 

(2) Revised program 

(3) Tests 


WHAT YOU WILL 
DO 


(1) Administer tests BEFORE and AFTER program 
administration 

(2) Administer program to sample of target audience 

(3) Score tests and identify proficiency levels 
attained 


FORMS YOU WILL 
USE 


FORM Je2(2) for recording test results. 



DESCRIPTION OF Sub-STEP 



INPUT 



Tests 
and 
revised 
instructional program 



J. 3.1 



ACTION 



-Administer and score 
before and after tests 
AND 

-Administer program to 
a large sample of 
target population 

ii 



OUTPUT 



Proficiency levels 
for sample 



iii 



Job Aid Contents 

CRITERIA FOR 

IDENTIFYING INPUTS ACTION TO BE TAKEN STANDARD FOR OUTPUTS FORMS TO USE 







-MATRIX: Adequacy 
of field test 
administration .229 


FORM J. 2(2) 
SUMMARY OF 

PROCEDURES . • . 228 



Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 


Final revised proyram 
plus tests 


J. 2. 5 






FORM J. 2 (2) 
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Sub- STEP I J. 3.1 \ 



JOB DIAGRAM 





INPUT 




ACTION 




OUTPUT 




Tests 
and 
revised 
instructional program 

i 


► 


-Administer and score 
AND 

-Attain ister program to 
a large sample of 
target population 

ii 




Proficiency levels 
for sample 








► 


iii 















Population of 
students" 



• a 



Seiect a 
random sample 
of students 
and administer 
tests and programs 



.a 



Rel iable evidence 
of program effectiveness 



I 1 1 .a 



Population of 
classes of students* 



i .b 



Select a 
random sampl e 

of classes 
and administer 
tests and programs 



M.b 



Rel I able evidence 
of program effectiveness 



1 i ( • b 



*One 2!1 '^^^ other of these approaches will be followed. 
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JOB PROCEDURES 





page 


SUMMARY HF PROCEDURES 


228 


Adequacy of the administration 
of the field test 


229 
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J. 3.1 



ILLUSTRATION SUMMARIZING PROCEDURES INVOLVED IN 
ADMINISTERING A FIELD TEST OF THE INSTRUCTIONAL PROGRAM* 



#1 

a. Select a sample 
of students which 
i s representat I ve 
of the target 
audience 

b. Whenever possible, 
select indivldus] 
students on 2 
random bas is 

c. If necessary to 
use Intact classes 
of students, 
select classes on 
a random basis (a 
mi nimum of about 
15 classes) 

d. Insure that the 
sample includes a 
minimum of approxi- 
mately 100 students. 
Sample sizes which 
are larger (up to 
approximately 3OO 
students) will yield 
more rel iable 

resul ts 



#2 

a. Administer criterion 
and sub-criterion 
tests as "before*' 
tests 

b. Administer 
instructional 
program 

c. Administer tests 
once again as an 
"dfter" test 

d. Score tests for 
errors 



a. 



n 

Record test 
resul ts on 
FORM J. 2(2) 



^Field results are likely to involve programs for more than one criterion behavior. 
They are likely to inalude programs for an entire course. 



FORM J. 2(2) 




228 



J.3.1 

STANDARDS 
MATRIX 



ADEQUACY OF ADM i N I STRATI ON OF FIELD TEST 



PROPERTIES 



SAMPLING ADEQUACY 



CRITERIA 



"Sample is randomly selected 
from tax* get audience: 

••Randomly by individual students 

• ^Randomly by classes of students 

-Sample size: 

•'Minvnum of 200 students 

•*Maximm of approximately 
300 students'^ 



COMPLETENESS 



-Tests are adninistered: 

••Before and after the 
administration of the program 

-Tlie program is administered to 
all (or nearly all) the sample ^ 
and all students are given 
adequate opportunity to complete 
the prograsTi 



*The more widespread the anticipated use of the program, the more desirable it is to 
have a large sample size for field testing the program. 
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PREVIEW OF THE NEXT SubSTEP 



YOUR PRODUCT 


An identification of percentage of studenio attaining 
' predetemined levels of proficiency (e.g^^ 90% 
correct) . 


WHAT YOU WILL 
WORK FROM 


(l) Test results 


WHAT YOU WILL 
DO 


(1) Compute percentage of students achieving each 
criterion beh-c*V!or 

(2) Compute percentage of criterion behaviors 
achieved 


FORMS YOU WILL 
USE 


None 



ERIC 



2^0 



DESCRIPTION OF Sub-STEP 



J. 3. 2 



] 





INPUT 




ACTION 




OUTPUT 




Test resiats 


► 


Ccimpute and record 
proficiency levels 
attained 




Identification of 
percentage of students 
attaining predetermined 
standards of proficiency 












► 






iv 




V 




vi 















Job Aid Contents 



CRITERIA FOR 
IDENTIFYING fNPUTS 



ACTION TO BE TAKEN STANDARD FOR OUTPUTS 



FORMS TO USE 



-MATRIX: Adequacy 
of report of 
results . - . . 235 



SUMMARY OF 
PROCEDURES 



23k 



Required Materials 



COMPLETED MATERIALS 

STEP 


COMPLETED FORMS 

STEP 


BLANK FORMS 






FORM J. 2. 2: Before 
and after test 
rfisul ts 


J. 3.1 
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Sub-STEP I J. 3.2 



JOB DIAGRAM 





1 niBi iir 
INPU 1 




ACTION 




OUTPUT 




Test results 


► 


Compute and record 
proficiency levels 
attained 


► 


Identification of 
percentage of students 
attaining predetermined 
standards of proficiency 


















iv 




V 




vi 















Number of 
students 
attaining a 
predetermi ned 
percentage of 
correct answers 
for each 
criterion behavior 



IV. a 



Compute and report 
the percentage 
of students 
attaining various 
proficiency levels 



v.a 



frequency distribution 
of percentages 
of students 
attaining various 
proficiency levels 

for each 
criterion behavior 



VI .a 



Number of 
criterion behaviors 

for which students 
attain predetermined 

proficiency levels 



iv.b 



Compute and report 
the percentage 
of criterion behavior 
attained by students 
at various 
proficiency levels 



v.b 



A frequency distribution 

of percentages 
of criterion behaviors 

for which students 
have attained varying 
proficiency levels 



vi .b 
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JOB PROCEDURES 





page 


SUMMARY OF PROCEDURES 




Adequacy of report of field test 


235 
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J. 3.2 



ILLUSTRATION SUMMARIZING PROCEDURES 
INVOLVED IN REPORTING FIELD TEST RESULTS 



»] 

From FORM J-2(2) 
summarizing test results 
for: 

••Each sub-criterion 
behavior related to 
each criterion 
behavior 

AND 

• •Cri terion behavior 

a. Compute for each 
student either: 

••The percentage of 
Items answered 
correctly 
OR 

••The percentage of 
items answered 
incorrectly 

b. Correct the number 

of students attaining 
proficiency (or 
errors if preferred) 
n various score 
ntervals : 

•90% or above 
•80%-39X 
•70?;-79^ 

c. Express the number of 
students attaining 
these levels as a 
percentage of the 
total sample 

d. Do this for "before" 
and "after" tests 



#2 

a. Report the number 
of sub-criterion 
behaviors associated 
wi th each cri terion 
behavior meeting 
various standards: 

• •Percentage of 
sub-cri terion 
behaviors for which 
(for example) 90^ 
of students attain 
90% of the test 
i terns 

b. Report other lesser 
standards met: 

••e.g., percentage of 
criterion behaviors 
for which 80% of 
students attain 30% 
of test items 

c. Present a frequency 
distribution of 
standards met for all 
criterion behaviors 
covered by an entire 
program 



a. Describe the sample 
of students used In 
the field test: 

• •Si ze of sample 

used 

• •Characteristics 

of sample: 

/Age 

/General ability 
levels 

/Other information 
which is aval 1- 
able and thouaht 
to be relevant 
to cri terion 
behaviors to be 
learned 
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J,3,3 

STANDARDS 
MATRIX 



ADEQUACY OF REPORT OF FIELD TEST RESULTS 



PROPERTIES 



COMPLETENESS 
of report of RESULTS 



COMPLETENESS 
of SAMPLE description 



CRITERIA 



-i4 report summarizing the 
percentage of students attaining 
various proficiency levels for: 

••Each sub-criterion behavior 
related to a criterion behavior 

••Each of the criterion hehaxncro 

••All of the criterion behaviors 
covered by a complete program 

-yl report summarizing the 
percentage of sub-criterion and 
criterion behaviors attained at 
varjfing levels of proficiency 

"A report summarizing the 
difference between results 
obtained '^before^' and ^'after'' 
program administration 



"A description of the field test 
sanple: 

••Ihmber of students 
• •How selected 
• *Age, IQ 

• •Other pertinent information 
available 
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STEP 



J. 3 



COMPLETION CHECKLIST 



J. 3.1 



lOENTIFlEO 


PERFORMED 


PRODUCED 


FORMS COMPLETED 




Administration of 
tests and program 

Scorino of tests 












A report of 

prof i ciency level s 





ERIC 



